Oct 30 21:39:36.562968 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:39:36.562977 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:39:36.562984 ip-10-0-131-231 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:39:36.563204 ip-10-0-131-231 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:39:47.725330 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:39:47.725344 ip-10-0-131-231 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3406883037974b639fbf4fd55ac4060d --
Oct 30 21:41:57.658075 ip-10-0-131-231 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:41:58.085710 ip-10-0-131-231 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:58.085710 ip-10-0-131-231 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:41:58.085710 ip-10-0-131-231 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:58.085710 ip-10-0-131-231 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094351    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094434    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094438    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094442    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094445    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094447    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094451    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.094444 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094453    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094456    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094459    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094462    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094465    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094468    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094471    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094473    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094476    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094479    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094482    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094485    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094488    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094490    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094493    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094497    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094499    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094502    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094505    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.094709 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094507    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094510    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094512    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094515    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094517    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094520    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094523    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094525    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094528    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094530    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094533    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094536    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094539    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094542    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094545    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094548    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094551    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094554    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094556    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094559    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.095252 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094561    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094564    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094566    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094569    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094572    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094574    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094577    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094580    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094582    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094585    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094588    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094590    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094593    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094595    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094598    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094600    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094603    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094606    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094608    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094611    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.095725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094614    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094616    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094619    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094622    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094624    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094628    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094632    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094634    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094637    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094640    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094642    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094645    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094649    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094652    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094654    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094657    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094660    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094664    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094667    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094669    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.096210 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094672    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094675    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094677    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094680    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094683    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094686    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094689    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094691    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094694    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094696    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094701    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094705    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094708    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094711    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094714    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094717    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094720    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094723    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094726    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.096679 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.094731    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095550    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095558    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095561    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095564    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095569    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095574    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095577    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095579    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095582    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095585    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095588    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095591    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095594    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095596    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095599    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095602    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095605    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095607    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.097175 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095610    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095612    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095615    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095617    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095620    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095622    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095627    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095630    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095632    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095635    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095638    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095640    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095643    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095646    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095649    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095651    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095653    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095656    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095659    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095661    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.097639 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095664    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095666    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095669    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095672    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095674    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095677    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095680    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095682    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095685    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095687    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095690    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095692    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095695    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095697    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095700    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095702    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095705    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095707    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095711    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.098122 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095714    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095716    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095719    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095722    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095724    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095727    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095730    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095733    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095736    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095738    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095741    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095744    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095746    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095749    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095751    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095754    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095757    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095761    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095764    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095768    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.098570 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095771    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095774    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095799    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095803    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095807    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095810    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095812    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095815    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095819    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095822    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095824    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095828    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095831    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095834    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095836    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095839    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095842    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095844    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095848    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095850    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.099060 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095853    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095856    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095858    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095861    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095863    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095866    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095869    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095871    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.095874    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096384    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096391    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096394    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096397    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096400    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096403    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096406    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096408    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096411    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096414    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.099556 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096416    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096421    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096423    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096426    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096429    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096432    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096435    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096437    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096442    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096446    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096449    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096452    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096455    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096457    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096460    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096463    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096465    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096468    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096470    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.100053 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096473    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096475    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096478    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096481    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096483    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096486    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096489    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096491    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096494    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096497    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096499    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096502    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096504    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096507    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096511    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096513    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096516    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096519    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096522    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.100501 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096525    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096527    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096530    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096532    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096535    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096537    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096540    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096542    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096545    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096548    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096550    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096553    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096555    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096558    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096560    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096562    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096565    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096567    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096570    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096572    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.101019 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096575    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096578    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096580    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096583    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096585    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096588    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096591    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096595    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096599    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096601    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096604    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096607    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096609    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096612    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096614    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096617    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096620    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096623    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096625    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096628    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.101485 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096631    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096633    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096636    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096638    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096641    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096643    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096646    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096648    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096651    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096653    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096656    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096659    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096661    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096664    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096667    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096669    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096674    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.096676    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097562    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097573    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:41:58.101994 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097581    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097586    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097590    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097594    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097598    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097602    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097605    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097608    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097612    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097615    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097618    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097621    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097624    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097627    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097630    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097633    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097638    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097641    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097645    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097648    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097651    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097655    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097659    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:41:58.102461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097662    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097665    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097668    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097672    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097675    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097678    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097681    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097685    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097688    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097695    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097698    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097701    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097704    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097712    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097715    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097718    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097721    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097725    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097733    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097736    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097739    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097742    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097745    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097748    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097751    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:41:58.103014 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097754    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097758    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097761    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097764    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097768    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097771    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097774    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097789    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097793    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097796    2578 flags.go:64] FLAG: --help="false"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097799    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097803    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097806    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097809    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097812    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097816    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097821    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097824    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097827    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097830    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097833    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097836    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097839    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097842    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:41:58.103626 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097845    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097848    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097851    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097854    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097857    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097860    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097863    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097869    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097872    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097875    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097878    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097881    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097884    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097887    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097890    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097895    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097898    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097902    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097905    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097908    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097912    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097915    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097918    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097921    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097925    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:41:58.104234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097932    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097935    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097938    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097941    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097944    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097947    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097950    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097953    2578 flags.go:64] FLAG: --port="10250"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097956    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097959    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ef5bfe7fe60adc8b"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097963    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097966    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097969    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097972    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097976    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097979    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097982    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097985    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097988    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097992    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097995    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.097998    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098001    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098004    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098007    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098010    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:41:58.104830 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098013    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098016    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098020    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098024    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098027    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098032    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098035    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098038    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098041    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098044    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098047    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098050    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098056    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098059    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098062    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098066    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098069    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098072    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098075    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098078    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098081    2578 flags.go:64] FLAG: --v="2"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098086    2578 flags.go:64] FLAG: --version="false"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098090    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098094    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098098    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:41:58.105462 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098254    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098258    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098262    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098265    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098268    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098270    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098273    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098276    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098278    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098281    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098286    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098288    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098293    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098295    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098298    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098300    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098303    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098306    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098308    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098311    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.106065 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098313    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098316    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098319    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098322    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098325    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098327    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098330    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098332    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098335    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098338    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098340    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098344    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098346    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098349    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098351    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098354    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098356    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098359    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098362    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098364    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.106532 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098367    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098369    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098373    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098376    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098381    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098383    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098386    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098389    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098391    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098394    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098396    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098399    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098402    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098404    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098407    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098409    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098412    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098415    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098417    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098420    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.107052 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098422    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098425    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098427    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098430    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098433    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098435    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098438    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098440    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098444    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098448    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098451    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098454    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098457    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098460    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098465    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098468    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098472    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098476    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098478    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.107519 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098481    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098484    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098486    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098489    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098491    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098494    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098496    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098499    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098502    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098504    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098507    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098510    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098512    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098515    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098517    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098520    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098524    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098527    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098530    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.107992 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098533    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098536    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098538    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098541    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098543    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098546    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098548    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098551    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.098579    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098617    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098622    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098627    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098631    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098636    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098639    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.108451 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098643    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098646    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098649    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098651    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098654    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098656    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098659    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098661    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098664    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098667    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098670    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098673    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098675    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098678    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098680    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098684    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098686    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098689    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098691    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098694    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.108818 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098697    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098700    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098702    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098705    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098707    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098711    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098714    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098716    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098720    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098724    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098726    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098729    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098732    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098734    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098737    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098739    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098743    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098745    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098748    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098750    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.109285 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098753    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098755    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098758    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098760    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098763    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098765    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098768    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098771    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098774    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098776    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098791    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098793    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098796    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098798    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098801    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098804    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098806    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098813    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098816    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098819    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.109755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098822    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098825    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098828    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098831    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098834    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098836    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098839    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098841    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098844    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098847    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098849    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098852    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098854    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098857    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098859    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098862    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098866    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098869    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098872    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.110232 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098875    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098877    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098880    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098883    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098885    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098888    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098891    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098893    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098896    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098914    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098920    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098923    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098926    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098930    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098933    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098935    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098939    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098942    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098945    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.110701 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098947    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.098950    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.105730    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.105744    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.106627    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.106633    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106688    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106694    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106697    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106701    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106704    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106707    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106710    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106713    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106716    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106718    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106721    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106724    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106726    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106729    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.111287 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106731    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106734    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106737    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106740    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106742    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106745    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106748    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106751    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106753    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106758    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106763    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106766    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106769    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106772    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106775    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106793    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106798    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106801    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106804    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.111734 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106807    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106809    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106812    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106815    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106818    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106820    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106823    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106826    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106829    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106831    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106834    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106837    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106839    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106842    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106844    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106847    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106849    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106852    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106855    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106858    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.112206 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106861    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106863    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106866    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106869    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106872    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106875    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106877    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106880    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106883    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106885    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106888    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106890    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106893    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106895    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106898    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106901    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106903    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106906    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106908    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106911    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.112666 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106914    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106916    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106919    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106921    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106924    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106926    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106929    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106932    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106934    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106937    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106940    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106943    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106945    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106948    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106951    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106953    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106957    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106959    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106962    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106965    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.113149 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106968    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106970    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106973    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106976    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106978    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106981    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106984    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106986    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106989    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106991    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106994    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.106996    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107000    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.107029    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107155    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107159    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.113609 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107162    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107166    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107169    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107171    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107174    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107177    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107179    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107182    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107184    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107189    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107193    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107196    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107199    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107202    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107205    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107208    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107211    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107214    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107217    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.113998 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107219    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107222    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107224    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107227    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107229    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107232    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107235    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107237    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107239    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107243    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107245    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107247    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107250    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107253    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107256    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107259    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107261    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107264    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107266    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107269    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.114437 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107272    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107274    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107277    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107279    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107282    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107285    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107287    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107290    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107293    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107295    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107298    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107300    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107303    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107305    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107308    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107310    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107313    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107315    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107318    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.114962 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107320    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107322    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107325    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107327    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107330    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107332    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107335    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107338    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107340    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107343    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107345    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107348    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107350    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107353    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107356    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107358    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107360    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107363    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107366    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107368    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.115402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107371    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107374    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107376    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107379    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107381    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107384    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107386    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107389    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107391    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107394    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107396    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107399    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107401    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107404    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107406    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107409    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107411    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107413    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107416    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.115874 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107419    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107421    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107425    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107429    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107432    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107437    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107439    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.107462    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107492    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107496    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107500    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107502    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107505    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107508    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107511    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107514    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:58.116326 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107517    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107519    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107522    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107525    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107528    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107531    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107534    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107536    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107539    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107541    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107544    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107546    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107549    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107551    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107554    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107556    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107559    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107561    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107564    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107566    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:58.116694 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107569    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107573    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107576    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107578    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107581    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107583    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107586    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107588    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107591    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107594    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107596    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107599    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107602    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107604    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107607    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107610    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107612    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107614    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107617    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:58.117168 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107619    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107622    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107625    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107627    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107630    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107633    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107635    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107638    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107640    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107643    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107645    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107648    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107650    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107655    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107659    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107662    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107665    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107667    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107670    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:58.117725 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107673    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107675    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107677    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107680    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107683    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107685    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107688    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107691    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107693    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107696    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107700    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107703    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107706    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107709    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107712    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107714    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107717    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107720    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107723    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:58.118475 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107725    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107727    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107730    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107732    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107736    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107738    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107741    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107743    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107747    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107749    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107752    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107754    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107756    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107759    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107761    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107764    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107766    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107769    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107771    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107773    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:58.119014 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:41:58.107776    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.107839    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.109837    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.110929    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.111028    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.111064    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.113182    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:58.119468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.114234    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:58.127957 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.127939    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:41:58.130955 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.130942    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 21:41:58.131732 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.131707    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:41:58.134331 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.134310    2578 fs.go:135] Filesystem UUIDs: map[120aae6e-3ed4-4ce9-bb93-092fea47ebc7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dc0b2a7b-df43-47cd-8cf0-005c4095efb3:/dev/nvme0n1p3]
Oct 30 21:41:58.134392 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.134329    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:41:58.139237 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.139133    2578 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:58.138654601 +0000 UTC m=+0.356822416 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099223 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb14ea987b096a0500ae82b33aaa9 SystemUUID:ec2bb14e-a987-b096-a050-0ae82b33aaa9 BootID:34068830-3797-4b63-9fbf-4fd55ac4060d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fc:01:46:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fc:01:46:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:c7:2e:37:2e:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:41:58.139332 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.139240    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:41:58.139332 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.139312    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:41:58.142824 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.142772    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:41:58.143038 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.142833    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-231.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:41:58.143081 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.143044    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:41:58.143081 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.143056    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:41:58.143132 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.143091    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:58.143861 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.143847    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:58.144535 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.144524    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:41:58.148157 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.148145    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:41:58.148277 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.148268    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:41:58.149236 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149226    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:41:58.149267 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149240    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:41:58.149267 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149250    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:41:58.149267 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149259    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:41:58.149398 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149270    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:41:58.149610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.149594    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:58.151956 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.151941    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:41:58.156569 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.156556    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:58.157671 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157659    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:41:58.157708 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157689    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:41:58.157875 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157864    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:41:58.157912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157881    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:41:58.157912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157887    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:41:58.157912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157893    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:41:58.157912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:41:58.157912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.157905    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:41:58.158611 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158602    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:41:58.158611 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158611    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:41:58.158672 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158619    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:41:58.158672 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158625    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:41:58.158672 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158639    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:41:58.158672 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.158648    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:41:58.160294 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.160284    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:41:58.160294 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.160294    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:41:58.164739 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.164723    2578 server.go:1263] "Started kubelet"
Oct 30 21:41:58.164839 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.164811    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:41:58.164945 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.164900    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:41:58.165038 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.164961    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:41:58.165572 ip-10-0-131-231 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:41:58.166714 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.165856    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:41:58.170062 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.170040    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:41:58.171264 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.171249    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:41:58.173970 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.173949    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:58.174443 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.174421    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:41:58.174604 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.174584    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:58.174987 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.174969    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:41:58.174987 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.174987    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:41:58.175133 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.174968    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:41:58.175133 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175096    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:41:58.175133 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175104    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:41:58.175752 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175730    2578 factory.go:153] Registering CRI-O factory
Oct 30 21:41:58.175752 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175749    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:41:58.175868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175814    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:41:58.175868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175824    2578 factory.go:55] Registering systemd factory
Oct 30 21:41:58.175868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175832    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:41:58.175868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175848    2578 factory.go:103] Registering Raw factory
Oct 30 21:41:58.175868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.175861    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:41:58.176868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.176854    2578 manager.go:319] Starting recovery of all containers
Oct 30 21:41:58.186621 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.186606    2578 manager.go:324] Recovery completed
Oct 30 21:41:58.191980 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.191964    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.193665 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.193634    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.193729 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.193684    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.193729 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.193700    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.194306 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.194296    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.194374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.194299    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:41:58.194374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.194328    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:41:58.194374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.194343    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:41:58.196404 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196389    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.196484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196414    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.196484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196423    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.196484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196435    2578 policy_none.go:50] "Start"
Oct 30 21:41:58.196484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196456    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:41:58.196484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.196470    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:41:58.198616 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.198603    2578 policy_none.go:44] "Start"
Oct 30 21:41:58.226406 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.226377    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:41:58.228742 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.228727    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:41:58.228822 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.228754    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:41:58.228822 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.228765    2578 server.go:87] "Starting device plugin registration server"
Oct 30 21:41:58.228988 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.228977    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:41:58.229030 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.228988    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:41:58.229156 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.229143    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:41:58.229234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.229224    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:41:58.229234 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.229233    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:41:58.229669 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.229641    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:41:58.229740 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.229685    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:58.310244 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.310222    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-231.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:41:58.310244 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.310225    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-231.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:41:58.311569 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.310363    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-231.us-west-2.compute.internal.18e36ca1a6a1085b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-231.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-231.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:58.164695131 +0000 UTC m=+0.382862944,LastTimestamp:2026-10-30 21:41:58.164695131 +0000 UTC m=+0.382862944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-231.us-west-2.compute.internal,}"
Oct 30 21:41:58.329660 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.329642    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.331789 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.331764    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.331881 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.331815    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.331881 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.331834    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.331881 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.331862    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.427543 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.427506    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:41:58.429439 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.428667    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:41:58.429439 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.428684    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:41:58.429439 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.428702    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:41:58.429439 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.428732    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:41:58.455072 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.455055    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:58.455161 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.455130    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zs528"
Oct 30 21:41:58.462633 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.462613    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-231.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:41:58.462723 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.462628    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:41:58.462723 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.462643    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:41:58.462860 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.462776    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-231.us-west-2.compute.internal.18e36ca1a85b15fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-231.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-231.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-231.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:58.193665533 +0000 UTC m=+0.411833356,LastTimestamp:2026-10-30 21:41:58.193665533 +0000 UTC m=+0.411833356,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-231.us-west-2.compute.internal,}"
Oct 30 21:41:58.515609 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.515585    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:58.516260 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.516242    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-231.us-west-2.compute.internal\" not found" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.529418 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.529389    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"]
Oct 30 21:41:58.529488 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.529455    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.530357 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.530343    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.530426 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.530370    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.530426 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.530381    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.532424 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.532413    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.532570 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.532557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.532610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.532589    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.533174 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533156    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.533263 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533163    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.533263 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533204    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.533263 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533214    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.533263 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533184    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.533263 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.533257    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.535413 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.535398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.535496 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.535428    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.536182 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.536163    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.536247 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.536194    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.536247 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.536205    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.565392 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.565371    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.567810 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.567792    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.578218 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.578198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.578306 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.578228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.578306 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.578252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.607803 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.607772    2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-231.us-west-2.compute.internal\" not found" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679401 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679401 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679401 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679576 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679442    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679576 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.679576 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.679473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.716468 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.716453    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:58.718145 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.718120    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:58.718217 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.718158    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:58.718217 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.718174    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:58.718217 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.718212    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.772239 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.772215    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.772239 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.772240    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-231.us-west-2.compute.internal\": node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:58.868142 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.868113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.871032 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:58.871018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:41:58.884812 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.884695    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:58.985413 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:58.985364    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.086450 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.086425    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.110291 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.110266    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:41:59.110482 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.110458    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:41:59.110522 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.110410    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a004d3cd47af64f05b1402e813782859-9b2f76aefe485d1f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-231.us-west-2.compute.internal.18e36ca1a85b7a7c\": read tcp 10.0.131.231:34268->3.22.246.75:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-231.us-west-2.compute.internal.18e36ca1a85b7a7c  default   5154 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-231.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-231.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-231.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:41:58 +0000 UTC,LastTimestamp:2026-10-30 21:41:58.530374774 +0000 UTC m=+0.748542587,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-231.us-west-2.compute.internal,}"
Oct 30 21:41:59.186944 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.186908    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.287505 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.287419    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.387616 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.387588    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.461644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.461607    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:58 +0000 UTC" deadline="2028-07-10 05:29:05.336678322 +0000 UTC"
Oct 30 21:41:59.461644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.461636    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14839h47m5.875045265s"
Oct 30 21:41:59.480551 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.480526    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:59.483872 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.483858    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:59.484873 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.484790    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:59.484873 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.484845    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:59.484873 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.484857    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:59.487965 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.487948    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.588084 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.588026    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.688752 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.688723    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.706861 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:41:59.706841    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:59.789914 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.789872    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.890505 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.890441    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:41:59.991263 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:41:59.991226    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:42:00.044931 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.044895    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:00.075022 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.074995    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:42:00.088151 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.088130    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:42:00.093403 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.093377    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e90b7cbbf3b5bb50f8f3af03ed08b90.slice/crio-973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2 WatchSource:0}: Error finding container 973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2: Status 404 returned error can't find the container with id 973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2
Oct 30 21:42:00.136059 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.136034    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:00.137222 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.137206    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:42:00.150208 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.150152    2578 apiserver.go:52] "Watching apiserver"
Oct 30 21:42:00.175311 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.175282    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:42:00.200912 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.200889    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:00.257212 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.257175    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:42:00.257508 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.257487    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ktk5z","openshift-multus/network-metrics-daemon-l84ww","kube-system/konnectivity-agent-9rtpn","kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal","openshift-dns/node-resolver-4zb2n","openshift-image-registry/node-ca-2cj8d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-ccs7w","openshift-ovn-kubernetes/ovnkube-node-t6z7q","kube-system/global-pull-secret-syncer-wr6rh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7","openshift-cluster-node-tuning-operator/tuned-nhnkd","openshift-multus/multus-7w2xx","openshift-network-diagnostics/network-check-target-68rh6"]
Oct 30 21:42:00.261469 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.261450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.263663 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.263635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.263781 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.263737    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:00.265913 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.265894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.268272 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.268251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.270545 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.270524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.273455 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.273435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.275784 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.275763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:00.275897 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.275849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:00.278198 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.278177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.280761 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.280735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.284133 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.284119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.284214 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.284191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:00.286580 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.286557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.287530 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287654 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287654 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287654 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jml4r\" (UniqueName: \"kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.287864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7cq\" (UniqueName: \"kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.287995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqr2\" (UniqueName: \"kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288256 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5kf\" (UniqueName: \"kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twznq\" (UniqueName: \"kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.288915 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.288623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.289303 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.289217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.318122 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.317878    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:42:00.318122 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.318028    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hdwjf\""
Oct 30 21:42:00.318122 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.318030    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.358142 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358116    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:42:00.358142 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358119    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2ds7\""
Oct 30 21:42:00.358142 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358118    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvvdp\""
Oct 30 21:42:00.358808 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358775    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.358808 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358804    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.358961 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358796    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:42:00.358961 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358910    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358974    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358987    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.358977    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.359013    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.359028    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:42:00.359063 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.359032    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:42:00.359771 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.359756    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:42:00.359833 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.359824    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.360037 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.360022    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:42:00.360622 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.360611    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.362510 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.362494    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:42:00.362572 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.362546    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:42:00.362616 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.362578    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.362759 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.362743    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwlkq\""
Oct 30 21:42:00.362831 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.362748    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6kl\""
Oct 30 21:42:00.364333 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.364319    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.364417 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.364404    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.369213 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369199    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.369279 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369256    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.369445 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369428    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.369524 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369475    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:42:00.369524 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369485    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-58hb8\""
Oct 30 21:42:00.369524 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369485    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rx46t\""
Oct 30 21:42:00.369646 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369529    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qk7p\""
Oct 30 21:42:00.369646 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369540    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:42:00.369646 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369575    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:42:00.369646 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369626    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jq2zx"
Oct 30 21:42:00.369773 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.369644    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48hj5\""
Oct 30 21:42:00.376506 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.376489    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:42:00.389025 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.389108 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.389108 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.389215 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.389215 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.389292 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.389292 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389239    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.389367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.389367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.389367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.389367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.389367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389408    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389491    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.389585 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4nt\" (UniqueName: \"kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.389837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389962    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.389993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.390093    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twznq\" (UniqueName: \"kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.390197 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.390208    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:00.890180494 +0000 UTC m=+3.108348319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390376    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshbk\" (UniqueName: \"kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390425    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390444    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jml4r\" (UniqueName: \"kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7cq\" (UniqueName: \"kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.390647 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj992\" (UniqueName: \"kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mffs\" (UniqueName: \"kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqr2\" (UniqueName: \"kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390971    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.390955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.391304 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.392110 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5kf\" (UniqueName: \"kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391508    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391595    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391835    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.392794 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.391960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392047    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392408    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.393340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.392518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.395433 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.395416    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.433594 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.433490    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerStarted","Data":"79516c8d3e2dff938854e1ffa92ef32523a106a5ee20984e522dd1f544b7005d"}
Oct 30 21:42:00.434514 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.434486    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"0e90b7cbbf3b5bb50f8f3af03ed08b90","Type":"ContainerStarted","Data":"973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2"}
Oct 30 21:42:00.448905 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.448876    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:00.448905 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.448904    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:00.449083 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.448919    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:00.449083 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.449006    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:00.948982014 +0000 UTC m=+3.167149835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:00.450791 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.450761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqr2\" (UniqueName: \"kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.450893 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.450814    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7cq\" (UniqueName: \"kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.450893 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.450823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5kf\" (UniqueName: \"kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.451282 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.451258    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twznq\" (UniqueName: \"kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.451321 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.451282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jml4r\" (UniqueName: \"kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.492882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.492882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493020    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.492980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493049 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493187    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.493317 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4nt\" (UniqueName: \"kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493400    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493421    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493505    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493534    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493585    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.493967 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493675    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493715    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493757    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493656    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshbk\" (UniqueName: \"kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493849    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.494652 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493997    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.493989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj992\" (UniqueName: \"kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494035    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mffs\" (UniqueName: \"kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494228    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.494258    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:00.495154 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.494315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:00.994299521 +0000 UTC m=+3.212467323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494334    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.495644 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.494500    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.496356 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.496333    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.496607 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.496580    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.497012 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.496994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.554069 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.554028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4nt\" (UniqueName: \"kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.554069 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.554042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mffs\" (UniqueName: \"kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.554069 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.554045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshbk\" (UniqueName: \"kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.554766 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.554747    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj992\" (UniqueName: \"kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.573039 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.573017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7w2xx"
Oct 30 21:42:00.575149 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.575127    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:00.579817 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.579777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:00.583500 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.583479    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ef60f7_bf87_4e70_bf23_4671a0ea4f23.slice/crio-3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934 WatchSource:0}: Error finding container 3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934: Status 404 returned error can't find the container with id 3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934
Oct 30 21:42:00.588102 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.588077    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb640ceb8_5397_4ff2_9428_ab03c9cf3474.slice/crio-41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a WatchSource:0}: Error finding container 41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a: Status 404 returned error can't find the container with id 41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a
Oct 30 21:42:00.598193 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.598169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:42:00.599299 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.599276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:42:00.604218 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.604203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:42:00.605935 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.605844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:42:00.610540 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.610518    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b36d9e_7722_4db1_acb4_7a75f0df705c.slice/crio-c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5 WatchSource:0}: Error finding container c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5: Status 404 returned error can't find the container with id c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5
Oct 30 21:42:00.616599 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.616571    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a0fce0_c116_4eb1_b233_066ba56adb42.slice/crio-d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4 WatchSource:0}: Error finding container d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4: Status 404 returned error can't find the container with id d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4
Oct 30 21:42:00.617457 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.617433    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf6b327_54b3_42dd_bf21_4c4ecd526af7.slice/crio-fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c WatchSource:0}: Error finding container fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c: Status 404 returned error can't find the container with id fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c
Oct 30 21:42:00.620402 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.620382    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1012b47f_8e91_41ca_bc3e_8597142bce89.slice/crio-21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466 WatchSource:0}: Error finding container 21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466: Status 404 returned error can't find the container with id 21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466
Oct 30 21:42:00.629548 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.629500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:42:00.631670 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.631651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:42:00.635810 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.635789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:00.639697 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.639673    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caeefbd_7ff9_4082_837d_48155be1d7b5.slice/crio-0acd6a8c4ec8f057dd624de3e3f164e7d4c98ccad5dd8b63cfc6e936c8bf3985 WatchSource:0}: Error finding container 0acd6a8c4ec8f057dd624de3e3f164e7d4c98ccad5dd8b63cfc6e936c8bf3985: Status 404 returned error can't find the container with id 0acd6a8c4ec8f057dd624de3e3f164e7d4c98ccad5dd8b63cfc6e936c8bf3985
Oct 30 21:42:00.644180 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.644151    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c9495d_4278_42d4_98dc_a2e8b1dc356d.slice/crio-03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647 WatchSource:0}: Error finding container 03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647: Status 404 returned error can't find the container with id 03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647
Oct 30 21:42:00.644782 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:00.644764    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c728bb_8670_4f08_a095_b0b9aa9adcc7.slice/crio-b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa WatchSource:0}: Error finding container b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa: Status 404 returned error can't find the container with id b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa
Oct 30 21:42:00.897642 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.897556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:00.897841 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.897679    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:00.897841 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.897744    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.89772845 +0000 UTC m=+4.115896267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:00.998976 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.998940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:00.998985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999101    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999114    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999133    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999145    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:00.999166 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999167    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.999152653 +0000 UTC m=+4.217320454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:00.999385 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:00.999195    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.999178919 +0000 UTC m=+4.217346736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:01.373108 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.372963    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:37:00 +0000 UTC" deadline="2028-07-01 23:15:19.389479725 +0000 UTC"
Oct 30 21:42:01.373108 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.373014    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14641h33m18.01646995s"
Oct 30 21:42:01.429579 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.429542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:01.429748 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:01.429681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:01.438725 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.438685    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ccs7w" event={"ID":"1012b47f-8e91-41ca-bc3e-8597142bce89","Type":"ContainerStarted","Data":"21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466"}
Oct 30 21:42:01.440340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.440311    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerStarted","Data":"fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c"}
Oct 30 21:42:01.441850 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.441825    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7" event={"ID":"f8c9495d-4278-42d4-98dc-a2e8b1dc356d","Type":"ContainerStarted","Data":"03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647"}
Oct 30 21:42:01.443372 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.443332    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cj8d" event={"ID":"67a0fce0-c116-4eb1-b233-066ba56adb42","Type":"ContainerStarted","Data":"d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4"}
Oct 30 21:42:01.444650 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.444627    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zb2n" event={"ID":"d7b36d9e-7722-4db1-acb4-7a75f0df705c","Type":"ContainerStarted","Data":"c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5"}
Oct 30 21:42:01.446039 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.446012    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rtpn" event={"ID":"b640ceb8-5397-4ff2-9428-ab03c9cf3474","Type":"ContainerStarted","Data":"41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a"}
Oct 30 21:42:01.447952 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.447926    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7w2xx" event={"ID":"69ef60f7-bf87-4e70-bf23-4671a0ea4f23","Type":"ContainerStarted","Data":"3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934"}
Oct 30 21:42:01.449629 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.449540    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa"}
Oct 30 21:42:01.450818 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.450797    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhnkd" event={"ID":"3caeefbd-7ff9-4082-837d-48155be1d7b5","Type":"ContainerStarted","Data":"0acd6a8c4ec8f057dd624de3e3f164e7d4c98ccad5dd8b63cfc6e936c8bf3985"}
Oct 30 21:42:01.909618 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:01.909577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:01.909835 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:01.909750    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:01.909835 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:01.909809    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:03.909790932 +0000 UTC m=+6.127958749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:02.010965 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.010926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:02.011167 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.010987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:02.011243 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011219    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:02.011243 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011237    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:02.011346 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011252    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:02.011346 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:04.011296398 +0000 UTC m=+6.229464203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:02.011469 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011392    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:02.011469 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.011429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:04.011416654 +0000 UTC m=+6.229584461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:02.374243 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.374138    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:37:00 +0000 UTC" deadline="2028-05-09 00:39:30.274746675 +0000 UTC"
Oct 30 21:42:02.374243 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.374174    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13346h57m27.900575775s"
Oct 30 21:42:02.433803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.433441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:02.433803 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.433595    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:02.433803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:02.433671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:02.433803 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:02.433743    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:03.429564 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:03.429528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:03.430251 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:03.429670    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:03.929196 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:03.929150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:03.929462 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:03.929357    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:03.929462 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:03.929427    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:07.929407243 +0000 UTC m=+10.147575048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:04.030506 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:04.030464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:04.030721 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:04.030521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:04.030721 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030697    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:04.030721 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030719    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:04.030933 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030734    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:04.030933 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:08.030776768 +0000 UTC m=+10.248944574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:04.030933 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030890    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:04.030933 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.030929    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:08.03091619 +0000 UTC m=+10.249083996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:04.433713 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:04.433680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:04.434172 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.433917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:04.434586 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:04.434560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:04.434772 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:04.434742    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:05.429346 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.429081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:05.429516 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:05.429458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:05.508463 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.508420    2578 generic.go:356] "Generic (PLEG): container finished" podID="77f8ada4fd8175df9d44ca0f297db617" containerID="97db5a62918c7a9b4545420a8b9f17beb8f0ee4355a0f5657236497a08ab5a78" exitCode=0
Oct 30 21:42:05.508979 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.508523    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerDied","Data":"97db5a62918c7a9b4545420a8b9f17beb8f0ee4355a0f5657236497a08ab5a78"}
Oct 30 21:42:05.517062 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.516343    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="61fba15671b56b70f82871a293ef59c91594acff794f22c9c58b581b9e141864" exitCode=0
Oct 30 21:42:05.517062 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.516428    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"61fba15671b56b70f82871a293ef59c91594acff794f22c9c58b581b9e141864"}
Oct 30 21:42:05.530687 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:05.530239    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"0e90b7cbbf3b5bb50f8f3af03ed08b90","Type":"ContainerStarted","Data":"d05bf1d5dcacebcfea8b44107abdd518de7461c5fce222469dc015cd82a83463"}
Oct 30 21:42:06.432091 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.432017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:06.432253 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:06.432162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:06.432326 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.432277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:06.432379 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:06.432363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:06.543501 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.543462    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rtpn" event={"ID":"b640ceb8-5397-4ff2-9428-ab03c9cf3474","Type":"ContainerStarted","Data":"7180f9b35b75eac7204bd3a4b7b2527b86821e88f304ab327e230886e3d3b63b"}
Oct 30 21:42:06.554072 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.549983    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerStarted","Data":"a16f14d5beb4c8cc445c1f560d6f920b0f3e072aab868dd8a10dd17a5821670a"}
Oct 30 21:42:06.554072 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.552478    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:06.554072 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.553565    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:06.666832 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.666755    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" podStartSLOduration=6.666734441 podStartE2EDuration="6.666734441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:05.944801586 +0000 UTC m=+8.162969414" watchObservedRunningTime="2026-10-30 21:42:06.666734441 +0000 UTC m=+8.884902268"
Oct 30 21:42:06.788730 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.788666    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rtpn" podStartSLOduration=2.473204938 podStartE2EDuration="7.788643406s" totalImagesPullingTime="5.315438468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:06.66859488 +0000 UTC m=+8.886762707" watchObservedRunningTime="2026-10-30 21:42:06.788643406 +0000 UTC m=+9.006811233"
Oct 30 21:42:06.915252 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:06.915187    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" podStartSLOduration=6.915164263 podStartE2EDuration="6.915164263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:06.788952681 +0000 UTC m=+9.007120509" watchObservedRunningTime="2026-10-30 21:42:06.915164263 +0000 UTC m=+9.133332089"
Oct 30 21:42:07.429400 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:07.429320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:07.429562 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:07.429460    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:07.555842 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:07.555805    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cj8d" event={"ID":"67a0fce0-c116-4eb1-b233-066ba56adb42","Type":"ContainerStarted","Data":"e76592bad1c0aee157d1052fcac8c2e79d48fe168204e1d463175a20d34357f6"}
Oct 30 21:42:07.682968 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:07.682879    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2cj8d" podStartSLOduration=2.482653291 podStartE2EDuration="8.682847726s" totalImagesPullingTime="6.200194435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:07.682082929 +0000 UTC m=+9.900250756" watchObservedRunningTime="2026-10-30 21:42:07.682847726 +0000 UTC m=+9.901015551"
Oct 30 21:42:07.965231 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:07.964872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:07.965231 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:07.965067    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:07.965231 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:07.965134    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:15.965109203 +0000 UTC m=+18.183277004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:08.065740 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.065705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:08.065939 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.065758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:08.065939 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.065902    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:08.065939 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.065927    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:08.065939 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.065939    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:08.066188 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.065996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.065978082 +0000 UTC m=+18.284145899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:08.066188 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.066083    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:08.066188 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.066117    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:16.066105278 +0000 UTC m=+18.284273092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:08.431291 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.431214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:08.431440 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.431332    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:08.431505 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.431493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:08.431620 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:08.431598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:08.560879 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.560829    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7" event={"ID":"f8c9495d-4278-42d4-98dc-a2e8b1dc356d","Type":"ContainerStarted","Data":"dc464fb798c128a754f25d632f3101e7981ed03a92b78b6deaaa6bd91804ac4b"}
Oct 30 21:42:08.561434 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:08.561414    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:09.429725 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:09.429689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:09.429919 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:09.429895    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:09.831836 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:09.831420    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:42:10.240764 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.240644    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:09.831436032Z","UUID":"d3ec1903-566f-423c-b663-3aa1b55c666a","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:42:10.242405 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.242383    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:42:10.242531 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.242413    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:42:10.430206 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.429834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:10.430206 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.429863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:10.430206 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:10.429986    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:10.430206 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:10.430137    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:10.568712 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.568640    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7" event={"ID":"f8c9495d-4278-42d4-98dc-a2e8b1dc356d","Type":"ContainerStarted","Data":"bbc492e26c06f8ec4de104ded1b84a80dc14b54eb7657d7263caa9a5c72423a9"}
Oct 30 21:42:10.570958 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.570918    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zb2n" event={"ID":"d7b36d9e-7722-4db1-acb4-7a75f0df705c","Type":"ContainerStarted","Data":"ba4d9d7ab5bd87e4e7fd48eaf454a466e8d80a478345702d199d92ca7999753b"}
Oct 30 21:42:10.573925 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.573894    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7w2xx" event={"ID":"69ef60f7-bf87-4e70-bf23-4671a0ea4f23","Type":"ContainerStarted","Data":"b20e8628c28979f8f41ee9f0964460cf96375da750edb815974bcd7b9fce967c"}
Oct 30 21:42:10.575900 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.575872    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerStarted","Data":"c771d27955abfd0d934ed1199e0b25aea3cafec433025f85761e2b70975045e3"}
Oct 30 21:42:10.684007 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.683966    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4zb2n" podStartSLOduration=2.60473689 podStartE2EDuration="11.683954106s" totalImagesPullingTime="9.079217216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:10.683761869 +0000 UTC m=+12.901929691" watchObservedRunningTime="2026-10-30 21:42:10.683954106 +0000 UTC m=+12.902121928"
Oct 30 21:42:10.914646 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:10.914386    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7w2xx" podStartSLOduration=2.9823372839999998 podStartE2EDuration="11.914369062s" totalImagesPullingTime="8.932031778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:10.914156629 +0000 UTC m=+13.132324451" watchObservedRunningTime="2026-10-30 21:42:10.914369062 +0000 UTC m=+13.132536884"
Oct 30 21:42:11.429479 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:11.429448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:11.429638 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:11.429575    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:11.583623 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:11.583589    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="c771d27955abfd0d934ed1199e0b25aea3cafec433025f85761e2b70975045e3" exitCode=0
Oct 30 21:42:11.583727 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:11.583690    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"c771d27955abfd0d934ed1199e0b25aea3cafec433025f85761e2b70975045e3"}
Oct 30 21:42:12.312577 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.312548    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:12.312976 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.312709    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:12.313128 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.313107    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:42:12.429668 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.429641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:12.429668 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.429652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:12.429892 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:12.429766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:12.429892 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:12.429870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:12.588894 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.588816    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7" event={"ID":"f8c9495d-4278-42d4-98dc-a2e8b1dc356d","Type":"ContainerStarted","Data":"95a6fa91404a2bb0943752416f0a64e45fa8621a613e0891bfe8ba7560eaf8c4"}
Oct 30 21:42:12.590458 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.590421    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nhnkd" event={"ID":"3caeefbd-7ff9-4082-837d-48155be1d7b5","Type":"ContainerStarted","Data":"00ff346c355c1b615362906eb7c0300a22278ed250eb3083017e07aed21719c8"}
Oct 30 21:42:12.591997 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.591975    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ccs7w" event={"ID":"1012b47f-8e91-41ca-bc3e-8597142bce89","Type":"ContainerStarted","Data":"94eceab2d1bdf431125ecbdb7323fe299f154bd10bf477dfd084455572aa3ec1"}
Oct 30 21:42:12.704036 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.703977    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7" podStartSLOduration=2.875072362 podStartE2EDuration="13.703959352s" totalImagesPullingTime="10.82888699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.703119732 +0000 UTC m=+14.921287548" watchObservedRunningTime="2026-10-30 21:42:12.703959352 +0000 UTC m=+14.922127177"
Oct 30 21:42:12.815834 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.815771    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nhnkd" podStartSLOduration=2.273154128 podStartE2EDuration="13.815758013s" totalImagesPullingTime="11.542603885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.815557556 +0000 UTC m=+15.033725378" watchObservedRunningTime="2026-10-30 21:42:12.815758013 +0000 UTC m=+15.033925835"
Oct 30 21:42:12.926960 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:12.926873    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ccs7w" podStartSLOduration=4.50390815 podStartE2EDuration="13.92685718s" totalImagesPullingTime="9.42294903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.926358618 +0000 UTC m=+15.144526440" watchObservedRunningTime="2026-10-30 21:42:12.92685718 +0000 UTC m=+15.145025002"
Oct 30 21:42:13.429662 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:13.429633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:13.430320 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:13.429763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:13.596247 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:13.596167    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="2190693cdfdaf19eb2dc7657a1ce1e42e39e86fa02f2c06167a98aa1a6487a28" exitCode=0
Oct 30 21:42:13.596407 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:13.596290    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"2190693cdfdaf19eb2dc7657a1ce1e42e39e86fa02f2c06167a98aa1a6487a28"}
Oct 30 21:42:14.432717 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:14.432691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:14.433423 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:14.432809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:14.433423 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:14.432820    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:14.433423 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:14.432909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:15.429470 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:15.429257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:15.429613 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:15.429562    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:15.603433 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:15.603319    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="e79536995a92969a90e26a3fdd8e67371fa5e65c11469d059a48d13e8a74443d" exitCode=0
Oct 30 21:42:15.603433 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:15.603376    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"e79536995a92969a90e26a3fdd8e67371fa5e65c11469d059a48d13e8a74443d"}
Oct 30 21:42:16.030745 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:16.030690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:16.030937 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.030875    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:16.030983 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.030946    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.030927232 +0000 UTC m=+34.249095036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:16.131056 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:16.131025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:16.131056 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:16.131059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:16.131234 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131164    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:16.131234 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131177    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:16.131234 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131186    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:16.131335 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131163    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:16.131335 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.131216568 +0000 UTC m=+34.349384368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:16.131335 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.131299    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.131285709 +0000 UTC m=+34.349453510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:16.429030 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:16.428954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:16.429173 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.429084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:16.429173 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:16.429130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:16.429271 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:16.429253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:17.001441 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.001206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.198759 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.198725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.409094 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.409028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.429122 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.429097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:17.429251 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:17.429208    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:17.594435 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.594400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:17.609842 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.609812    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"408ced0064f33fcbe48649ec78fd909f3ae50a9ed9a0cc537bdbc0054a180b72"}
Oct 30 21:42:17.609947 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.609903    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"0e98e87f2dfe260e2f718298ff84458ddada2f15c75ddbb07bf9353eda6fe089"}
Oct 30 21:42:17.609947 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.609937    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"48f6eded7f8fc32e5af5c97f29a73f197a8fdabb916e8c5d610ddb56fa5ae0a4"}
Oct 30 21:42:17.610024 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.609975    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"185544fd457b74076fa41c8cf8b17534712936bac0f7574f944bac5289cf646d"}
Oct 30 21:42:17.610024 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.610002    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"bc66094d6dcbfbb4b8d417e6a566877c88b6cd1b728f35820268d5df94843699"}
Oct 30 21:42:17.610079 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.610031    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"dbd64459e47404361ccbb6482925a681e389c2005abe114e18b82108d185f266"}
Oct 30 21:42:17.761718 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:17.761687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:18.429965 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.429927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:18.430398 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:18.430026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:18.430398 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.430076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:18.430398 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:18.430136    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:18.614609 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.614574    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="4525b04a7a01f6ac9b55ee82c4e7c18e6526e0352c228f11c53bdf2a0cd2face" exitCode=0
Oct 30 21:42:18.614737 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.614633    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"4525b04a7a01f6ac9b55ee82c4e7c18e6526e0352c228f11c53bdf2a0cd2face"}
Oct 30 21:42:18.711642 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.711522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:18.859485 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:18.859451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.057959 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.057927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.219290 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.219262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:19.429751 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.429675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:19.429902 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:19.429766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:19.619801 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.619753    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"1845d6e9319b55711dda9ee947b674f128e9bb622b0813bb97ff2cf05f883453"}
Oct 30 21:42:19.625420 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.625392    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="5cbc42e8b5d6f93ef11e7502229b46d1d1d225e415aa77f0e193ab0479a5152d" exitCode=0
Oct 30 21:42:19.625546 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:19.625431    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerDied","Data":"5cbc42e8b5d6f93ef11e7502229b46d1d1d225e415aa77f0e193ab0479a5152d"}
Oct 30 21:42:20.428973 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:20.428943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:20.429170 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:20.429045    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:20.429170 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:20.429096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:20.429246 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:20.429193    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:20.630615 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:20.630586    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" event={"ID":"5bf6b327-54b3-42dd-bf21-4c4ecd526af7","Type":"ContainerStarted","Data":"0639a566ac182f8324f8c8d5c39130408b09f6fd78a5ba0acacedf01fd7edef4"}
Oct 30 21:42:21.092849 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.092821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:21.249605 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.249570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:21.429312 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.429233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:21.429440 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:21.429335    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:21.636484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.636447    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" event={"ID":"f7c728bb-8670-4f08-a095-b0b9aa9adcc7","Type":"ContainerStarted","Data":"15c88a9ec3d50a74f77b4aa5bf0399c33da3fc43ed286bb7b8aa118864d59c02"}
Oct 30 21:42:21.901077 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.900876    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ktk5z" podStartSLOduration=8.899190672 podStartE2EDuration="22.900859192s" totalImagesPullingTime="14.00166852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.746552441 +0000 UTC m=+22.964720262" watchObservedRunningTime="2026-10-30 21:42:21.900859192 +0000 UTC m=+24.119027013"
Oct 30 21:42:21.901346 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.901321    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q" podStartSLOduration=6.985972273 podStartE2EDuration="22.90131455s" totalImagesPullingTime="15.915342277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.853037453 +0000 UTC m=+24.071205276" watchObservedRunningTime="2026-10-30 21:42:21.90131455 +0000 UTC m=+24.119482371"
Oct 30 21:42:21.918842 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:21.918812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.091864 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.091837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.245852 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.245823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.411564 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.411532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.429818 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.429798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:22.429908 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.429828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:22.429939 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:22.429909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:22.430083 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:22.430055    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:22.639300 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.639218    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:22.639733 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.639304    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:22.639733 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.639354    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:22.654356 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.654334    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:22.654469 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.654437    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:22.694726 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.694701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:22.844569 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:22.844539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:23.002949 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.002922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:23.161384 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.161358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:23.429640 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.429567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:23.429774 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:23.429670    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:23.873878 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.873848    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wr6rh"]
Oct 30 21:42:23.874280 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.873942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:23.874280 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:23.874030    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:23.877467 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.877438    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68rh6"]
Oct 30 21:42:23.877590 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.877527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:23.877657 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:23.877614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:23.878060 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.878040    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l84ww"]
Oct 30 21:42:23.878169 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:23.878133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:23.878252 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:23.878233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:24.173726 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:24.173639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:24.343386 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:24.343356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:24.933383 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:24.933354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:25.429563 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:25.429535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:25.429745 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:25.429618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:25.429745 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:25.429627    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:25.429745 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:25.429699    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:25.429745 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:25.429736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:25.429880 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:25.429818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:26.047241 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:26.047206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:26.485161 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:26.485132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:26.660750 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:26.660718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:26.834114 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:26.834046    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4zb2n_d7b36d9e-7722-4db1-acb4-7a75f0df705c/dns-node-resolver/0.log"
Oct 30 21:42:26.883900 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:26.883872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:27.100972 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.100876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:27.274552 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.274496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:27.429143 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.429058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:27.429316 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.429067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:27.429316 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:27.429187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:27.429316 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.429195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:27.429316 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:27.429248    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:27.429316 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:27.429278    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:27.516506 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.516483    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cj8d_67a0fce0-c116-4eb1-b233-066ba56adb42/node-ca/0.log"
Oct 30 21:42:27.671774 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.671707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:27.850635 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:27.850598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.347505 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:28.347472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.565706 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:28.565673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.750897 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:28.750865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:28.918817 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:28.918773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.075300 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.075224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.332991 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.332911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.429156 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.429123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:29.429156 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.429153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:29.429605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:29.429270    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:29.429605 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.429312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:29.429605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:29.429384    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:29.429605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:29.429468    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:29.601969 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.601884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:29.760943 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:29.760913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.144867 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:30.144836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.331126 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:30.331086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.590343 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:30.590316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.758220 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:30.758189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:30.922388 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:30.922304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.126368 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:31.126336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:31.429218 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:31.429188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:31.429218 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:31.429213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:31.429404 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:31.429286    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:31.429404 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:31.429319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:31.429404 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:31.429380    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:31.429510 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:31.429458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:31.663405 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:31.663373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.045401 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.045359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:32.045574 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.045496    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:32.045574 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.045553    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs podName:099cdf15-a1f3-456c-bbf5-34b2363b6142 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:04.045539022 +0000 UTC m=+66.263706823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs") pod "network-metrics-daemon-l84ww" (UID: "099cdf15-a1f3-456c-bbf5-34b2363b6142") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:32.146444 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.146413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.146449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146549    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146572    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146591    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146603    2578 projected.go:196] Error preparing data for projected volume kube-api-access-4dslv for pod openshift-network-diagnostics/network-check-target-68rh6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:32.146605 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146607    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret podName:54348157-31ac-475e-90a1-32b963be27a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:04.146589488 +0000 UTC m=+66.364757290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret") pod "global-pull-secret-syncer-wr6rh" (UID: "54348157-31ac-475e-90a1-32b963be27a3") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:32.146832 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:32.146646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv podName:7e23aa0b-cefb-4c96-b4a6-d48235deb03b nodeName:}" failed. No retries permitted until 2026-10-30 21:43:04.146631793 +0000 UTC m=+66.364799613 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dslv" (UniqueName: "kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv") pod "network-check-target-68rh6" (UID: "7e23aa0b-cefb-4c96-b4a6-d48235deb03b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:32.322025 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.321959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.534765 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.534736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.729699 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.729670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:32.925487 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:32.925451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.128975 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.128903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.429275 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.429206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:33.429275 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.429250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:33.429441 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:33.429306    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:33.429441 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:33.429347    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:33.429441 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.429381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:33.432060 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:33.429761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:33.706907 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.706819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:33.863900 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:33.863868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.333022 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:34.332997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.521592 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:34.521538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.723065 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:34.723014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:34.958963 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:34.958930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.130240 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:35.130168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:35.429149 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:35.429065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:35.429149 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:35.429107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:35.429361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:35.429166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:35.429361 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:35.429182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:35.429361 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:35.429242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:35.429361 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:35.429311    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:36.126689 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:36.126645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.319532 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:36.319497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.521529 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:36.521499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:36.779239 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:36.779136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:37.429881 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:37.429853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:37.430367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:37.429888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:37.430367 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:37.429969    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:37.430367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:37.429859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:37.430367 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:37.430252    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:37.430566 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:37.430389    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:37.847721 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:37.847691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:39.429490 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:39.429441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:39.429490 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:39.429492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:39.430077 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:39.429602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:39.430077 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:39.429618    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:39.430077 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:39.429687    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:39.430077 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:39.429763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:40.581369 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:40.581336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:40.749483 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:40.749451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:40.913842 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:40.913725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.089262 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.089238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.279790 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.279743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.429735 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.429702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:41.429735 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.429734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:41.429929 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.429829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:41.429929 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:41.429834    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:41.429929 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:41.429913    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:41.430036 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:41.430005    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:41.441633 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.441590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.604595 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.604501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:41.803689 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:41.803649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.239666 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:42.239635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.442623 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:42.442593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.601358 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:42.601263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.767330 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:42.767303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:42.920557 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:42.920477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:43.428925 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.428895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:43.429112 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.428895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:43.429112 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:43.428998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l84ww" podUID="099cdf15-a1f3-456c-bbf5-34b2363b6142"
Oct 30 21:42:43.429112 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:43.429053    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68rh6" podUID="7e23aa0b-cefb-4c96-b4a6-d48235deb03b"
Oct 30 21:42:43.429112 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.428901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:43.429266 ip-10-0-131-231 kubenswrapper[2578]: E1030 21:42:43.429135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wr6rh" podUID="54348157-31ac-475e-90a1-32b963be27a3"
Oct 30 21:42:43.430103 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.430082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:43.529988 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.529960    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:42:43.530140 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.530089    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:42:43.597279 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.597246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:43.639385 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.639364    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9zbm"]
Oct 30 21:42:43.663579 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.663558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.686927 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.686874    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-79gfb"]
Oct 30 21:42:43.705720 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.705700    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dmnk2"]
Oct 30 21:42:43.705845 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.705828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.713283 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.713261    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:43.713374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.713294    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:43.713374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.713297    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ssgdc\""
Oct 30 21:42:43.713374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.713337    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:43.717496 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.717482    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9zbm"]
Oct 30 21:42:43.717545 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.717507    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79gfb"]
Oct 30 21:42:43.717584 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.717574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.727493 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.727476    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmnk2"]
Oct 30 21:42:43.801538 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.801508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:43.814097 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814046    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:43.814237 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814131    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:43.814237 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814149    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:43.814237 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814190    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:43.814237 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814191    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:43.814430 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814292    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxkz7\""
Oct 30 21:42:43.814430 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814299    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j86j7\""
Oct 30 21:42:43.814430 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.814312    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:43.828717 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.828837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.828837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.828989 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.828989 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.828989 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnxl\" (UniqueName: \"kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.829106 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.828998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z99j\" (UniqueName: \"kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.829106 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.829058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.829106 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.829083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.829106 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.829098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4s5\" (UniqueName: \"kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.829276 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.829189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.930219 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.930361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.930361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930361 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnxl\" (UniqueName: \"kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z99j\" (UniqueName: \"kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4s5\" (UniqueName: \"kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.930610 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930923 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.930923 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930905    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.931016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.930962    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.931137 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.931117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.933919 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.933897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.934023 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.933904    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.934023 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.933904    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.955552 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.955485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:43.985332 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.985307    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z99j\" (UniqueName: \"kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:43.985572 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.985555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4s5\" (UniqueName: \"kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:43.985821 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.985807    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnxl\" (UniqueName: \"kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:43.988599 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:43.988585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:42:44.052957 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.052931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:42:44.080985 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.080962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:44.268456 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.268427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.424051 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.424026    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9zbm"]
Oct 30 21:42:44.426912 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:44.426883    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6dc066_7ed1_4e73_86ad_5455718c1106.slice/crio-7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6 WatchSource:0}: Error finding container 7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6: Status 404 returned error can't find the container with id 7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6
Oct 30 21:42:44.432829 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.432805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:44.471999 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.471974    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79gfb"]
Oct 30 21:42:44.474387 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:44.474367    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465b6111_ca93_426c_9901_e452be8ba99e.slice/crio-37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd WatchSource:0}: Error finding container 37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd: Status 404 returned error can't find the container with id 37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd
Oct 30 21:42:44.477568 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.477505    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dmnk2"]
Oct 30 21:42:44.481288 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:44.481245    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a8572a_f8ad_4c98_8758_cc25c4823aaa.slice/crio-70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d WatchSource:0}: Error finding container 70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d: Status 404 returned error can't find the container with id 70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d
Oct 30 21:42:44.684036 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.684005    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmnk2" event={"ID":"c1a8572a-f8ad-4c98-8758-cc25c4823aaa","Type":"ContainerStarted","Data":"70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d"}
Oct 30 21:42:44.685380 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.685360    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79gfb" event={"ID":"465b6111-ca93-426c-9901-e452be8ba99e","Type":"ContainerStarted","Data":"6b65420d05340cbc62697c2ab118555fd735aab8e784a2d39e3c2489e8a8a5d2"}
Oct 30 21:42:44.685480 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.685401    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79gfb" event={"ID":"465b6111-ca93-426c-9901-e452be8ba99e","Type":"ContainerStarted","Data":"37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd"}
Oct 30 21:42:44.686372 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:44.686352    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9zbm" event={"ID":"cf6dc066-7ed1-4e73-86ad-5455718c1106","Type":"ContainerStarted","Data":"7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6"}
Oct 30 21:42:45.259763 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.259725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:45.415188 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.415153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:45.429105 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.429080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:42:45.429105 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.429103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:42:45.429249 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.429226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:42:45.479166 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.479141    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:45.526887 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526839    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:45.526887 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526823    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s2r8\""
Oct 30 21:42:45.527013 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526901    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hmtv9\""
Oct 30 21:42:45.527013 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526919    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:45.527013 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526933    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:45.527013 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.526996    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8c5b8\""
Oct 30 21:42:45.659528 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.659489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:45.821446 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.821422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:45.984908 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:45.984874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:46.154138 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.154060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:46.203286 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.203251    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"]
Oct 30 21:42:46.205290 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.205266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.256703 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.256676    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:42:46.303178 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.303145    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:42:46.303618 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.303294    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-27snf\""
Oct 30 21:42:46.303618 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.303334    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:42:46.304326 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.304274    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"]
Oct 30 21:42:46.304457 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.304390    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:42:46.304457 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.304416    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:42:46.307496 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.307459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:46.347956 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.347926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.348087 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.347967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.348087 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.347991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887zx\" (UniqueName: \"kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.348258 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.348217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.449057 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.448989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.449057 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.449053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.449278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.449072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.449278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.449093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887zx\" (UniqueName: \"kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.450515 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.450480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.452189 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.452171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.456204 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.456175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.506030 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.505992    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887zx\" (UniqueName: \"kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.517611 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.517585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:42:46.693893 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.693852    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79gfb" event={"ID":"465b6111-ca93-426c-9901-e452be8ba99e","Type":"ContainerStarted","Data":"ae60c3c5100904f2d4c0ee5922a9594fb3d29016211484f91094d21e5cc98e08"}
Oct 30 21:42:46.884674 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:46.884649    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"]
Oct 30 21:42:46.922026 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:46.921983    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b30f97_b15d_4c82_b314_2849193c3a98.slice/crio-d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd WatchSource:0}: Error finding container d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd: Status 404 returned error can't find the container with id d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd
Oct 30 21:42:47.698612 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:47.698571    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" event={"ID":"e8b30f97-b15d-4c82-b314-2849193c3a98","Type":"ContainerStarted","Data":"d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd"}
Oct 30 21:42:47.700820 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:47.700772    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmnk2" event={"ID":"c1a8572a-f8ad-4c98-8758-cc25c4823aaa","Type":"ContainerStarted","Data":"d4e4bcec58dcb261a17d1c72512d7fde17d9ad4a2077b4631fe08354e91b956b"}
Oct 30 21:42:47.700945 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:47.700847    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dmnk2" event={"ID":"c1a8572a-f8ad-4c98-8758-cc25c4823aaa","Type":"ContainerStarted","Data":"4f64ced258a7ba7ffd3445a485f977d0e1f571315975f250007fc6dad140258e"}
Oct 30 21:42:47.701476 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:47.701457    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:47.818089 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:47.818026    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dmnk2" podStartSLOduration=2.42880563 podStartE2EDuration="4.81801089s" totalImagesPullingTime="2.38920526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:47.817919039 +0000 UTC m=+50.036086868" watchObservedRunningTime="2026-10-30 21:42:47.81801089 +0000 UTC m=+50.036178709"
Oct 30 21:42:48.705289 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:48.705218    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79gfb" event={"ID":"465b6111-ca93-426c-9901-e452be8ba99e","Type":"ContainerStarted","Data":"10008fa792b4fe2b5e2bf6603642debaa9c436bc3c769058288eda74b3e21045"}
Oct 30 21:42:48.706691 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:48.706670    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9zbm" event={"ID":"cf6dc066-7ed1-4e73-86ad-5455718c1106","Type":"ContainerStarted","Data":"b66ad250c4fe0687bef454dbe93722f56d1464555a1c7c81289ae25d243f6ad0"}
Oct 30 21:42:48.709916 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:48.709893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:48.898172 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:48.898132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:49.074484 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.074457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:49.347488 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.347405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:49.548935 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.548902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:49.713225 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.713190    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" event={"ID":"e8b30f97-b15d-4c82-b314-2849193c3a98","Type":"ContainerStarted","Data":"6124b69cee66d923a98f70afdc18e8879f37072a1b87c27b5bd95ce27810bda7"}
Oct 30 21:42:49.713625 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.713250    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" event={"ID":"e8b30f97-b15d-4c82-b314-2849193c3a98","Type":"ContainerStarted","Data":"d4eaf1cca0152e83d61820b50e91d40b052b30a1ddc47bc60b5b642bc33d8920"}
Oct 30 21:42:49.797696 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.797652    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-79gfb" podStartSLOduration=3.48802371 podStartE2EDuration="6.797638955s" totalImagesPullingTime="3.309615245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:49.797403985 +0000 UTC m=+52.015571807" watchObservedRunningTime="2026-10-30 21:42:49.797638955 +0000 UTC m=+52.015806768"
Oct 30 21:42:49.797953 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.797925    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9zbm" podStartSLOduration=2.885881418 podStartE2EDuration="6.797918747s" totalImagesPullingTime="3.912037329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:49.685174204 +0000 UTC m=+51.903342027" watchObservedRunningTime="2026-10-30 21:42:49.797918747 +0000 UTC m=+52.016086568"
Oct 30 21:42:49.915871 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:49.915823    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" podStartSLOduration=2.100241193 podStartE2EDuration="3.915808773s" totalImagesPullingTime="1.81556758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:49.91521275 +0000 UTC m=+52.133380570" watchObservedRunningTime="2026-10-30 21:42:49.915808773 +0000 UTC m=+52.133976592"
Oct 30 21:42:50.078535 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:50.078466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:50.244535 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:50.244506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:50.493283 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:50.493214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:50.662893 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:50.662866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:51.879867 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:51.879825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.067665 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.067635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.239480 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.239451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.409097 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.409067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.576245 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.576167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.747870 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.747829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:52.867143 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.867072    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-65jnb"]
Oct 30 21:42:52.870151 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.870135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:52.915897 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:52.915869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.005614 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd48\" (UniqueName: \"kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.005739 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.005739 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.005868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.005868 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.005960 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.006013 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.005989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.006082 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.006027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.006082 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.006057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.029337 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.029313    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:42:53.029642 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.029626    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv9kv\""
Oct 30 21:42:53.029704 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.029669    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:42:53.029704 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.029696    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:42:53.106879 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107016 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.106995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107205 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd48\" (UniqueName: \"kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107205 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107059    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107205 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107205 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107205 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107148    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107430 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107430 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107529 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.107631 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.107572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.109149 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.109128    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.109300 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.109281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.157843 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.157753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.162449 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.162427    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd48\" (UniqueName: \"kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.184211 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.184187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:42:53.194930 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:53.194907    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a1da5e_7477_453d_bd89_cbe7f9e7b20a.slice/crio-740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b WatchSource:0}: Error finding container 740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b: Status 404 returned error can't find the container with id 740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b
Oct 30 21:42:53.417174 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.417099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:53.726538 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:53.726503    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65jnb" event={"ID":"76a1da5e-7477-453d-bd89-cbe7f9e7b20a","Type":"ContainerStarted","Data":"740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b"}
Oct 30 21:42:54.091654 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.091581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:54.355590 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.355512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:54.639050 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.638951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:54.657015 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.656813    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:42:54.731079 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.731057    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65jnb" event={"ID":"76a1da5e-7477-453d-bd89-cbe7f9e7b20a","Type":"ContainerStarted","Data":"d4738e8301ef32a26b1bdb8cf9edb76036801ee385ed3be0156b7778d91f6a25"}
Oct 30 21:42:54.791898 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:54.791867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.012476 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.012448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.173501 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.173472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:55.390568 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.390491    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"]
Oct 30 21:42:55.393735 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.393720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425568 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425701 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425701 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425701 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.425886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.425819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lqd\" (UniqueName: \"kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.443833 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.443807    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:42:55.443929 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.443912    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:42:55.443979 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.443932    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:42:55.444101 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.444086    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ag5tqi2uauu0p\""
Oct 30 21:42:55.444140 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.444105    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:42:55.444140 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.444109    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:42:55.444140 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.444131    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k8pd4\""
Oct 30 21:42:55.490949 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.490926    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"]
Oct 30 21:42:55.526705 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.526838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.526838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.526838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.526838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lqd\" (UniqueName: \"kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527050 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527050 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527050 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.526947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527050 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.527009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527227 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.527175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.527499 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.527473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.529480 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.529565 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529490    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.529736 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529718    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.529845 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.529965 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.530024 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.529977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.582204 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.582185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lqd\" (UniqueName: \"kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.714977 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.714955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:42:55.736264 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.736237    2578 generic.go:356] "Generic (PLEG): container finished" podID="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" containerID="d4738e8301ef32a26b1bdb8cf9edb76036801ee385ed3be0156b7778d91f6a25" exitCode=0
Oct 30 21:42:55.736364 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:55.736281    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65jnb" event={"ID":"76a1da5e-7477-453d-bd89-cbe7f9e7b20a","Type":"ContainerDied","Data":"d4738e8301ef32a26b1bdb8cf9edb76036801ee385ed3be0156b7778d91f6a25"}
Oct 30 21:42:56.126487 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.126396    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"]
Oct 30 21:42:56.128922 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:56.128882    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9185dde_8443_434d_9df7_27d7e287b372.slice/crio-c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39 WatchSource:0}: Error finding container c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39: Status 404 returned error can't find the container with id c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39
Oct 30 21:42:56.740250 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.740220    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39"}
Oct 30 21:42:56.742312 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.742288    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65jnb" event={"ID":"76a1da5e-7477-453d-bd89-cbe7f9e7b20a","Type":"ContainerStarted","Data":"44addc5998ba7d7688ac2cd87cb37cbc29b35884d76749b20709b85fd0aa5840"}
Oct 30 21:42:56.742427 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.742329    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-65jnb" event={"ID":"76a1da5e-7477-453d-bd89-cbe7f9e7b20a","Type":"ContainerStarted","Data":"2beb6e41f0a98939542d1a6ac4c4bfadc046633e6f91a186dcba8043a2fe7f0c"}
Oct 30 21:42:56.785571 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.785539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:56.855453 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.855401    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-65jnb" podStartSLOduration=3.3117724 podStartE2EDuration="4.855388055s" totalImagesPullingTime="1.479046515s" totalInitContainerRuntime="64.56914ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:56.855171509 +0000 UTC m=+59.073339327" watchObservedRunningTime="2026-10-30 21:42:56.855388055 +0000 UTC m=+59.073555893"
Oct 30 21:42:56.944410 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:56.944374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:57.112069 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.111970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:57.277334 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.277298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:57.449606 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.449526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:57.544336 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.544295    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"]
Oct 30 21:42:57.548714 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.548457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.602144 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.602099    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:42:57.603086 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.603054    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"]
Oct 30 21:42:57.608274 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.607329    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k4gxb\""
Oct 30 21:42:57.640921 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.640889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.641075 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.641049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9c2\" (UniqueName: \"kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.642747 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.642726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:57.741521 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.741491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9c2\" (UniqueName: \"kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.741979 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.741554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.744621 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.744593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.797818 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.797793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9c2\" (UniqueName: \"kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.862727 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.862693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:42:57.900134 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:57.900091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:58.230391 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.230360    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"]
Oct 30 21:42:58.234755 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:42:58.234732    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8d50c8_864a_4758_9ece_62c693d9cffd.slice/crio-a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea WatchSource:0}: Error finding container a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea: Status 404 returned error can't find the container with id a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea
Oct 30 21:42:58.635070 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.635006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:58.715756 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.715733    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:42:58.751264 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.751238    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz" event={"ID":"ab8d50c8-864a-4758-9ece-62c693d9cffd","Type":"ContainerStarted","Data":"a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea"}
Oct 30 21:42:58.753188 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.753163    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"b85b151712d8239b4fa32591e8e46e74538548f1a50d94e49bc82ef3d60f5e20"}
Oct 30 21:42:58.753269 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.753233    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"2724879fac90425eccd16b230da09bf24fb66dcc97191214d3865b2c97a5b272"}
Oct 30 21:42:58.796367 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.796344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:58.952300 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:58.952232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:42:59.760283 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:59.760255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:42:59.760744 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:59.760293    2578 generic.go:356] "Generic (PLEG): container finished" podID="e8b30f97-b15d-4c82-b314-2849193c3a98" containerID="d4eaf1cca0152e83d61820b50e91d40b052b30a1ddc47bc60b5b642bc33d8920" exitCode=1
Oct 30 21:42:59.760744 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:59.760384    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" event={"ID":"e8b30f97-b15d-4c82-b314-2849193c3a98","Type":"ContainerDied","Data":"d4eaf1cca0152e83d61820b50e91d40b052b30a1ddc47bc60b5b642bc33d8920"}
Oct 30 21:42:59.760899 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:59.760813    2578 scope.go:122] "RemoveContainer" containerID="d4eaf1cca0152e83d61820b50e91d40b052b30a1ddc47bc60b5b642bc33d8920"
Oct 30 21:42:59.762837 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:42:59.762809    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"ba15fb306bd43be15460949338d66f85c24088ff9ca8156da7d7e9c9b16204d6"}
Oct 30 21:43:00.767923 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.767889    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz" event={"ID":"ab8d50c8-864a-4758-9ece-62c693d9cffd","Type":"ContainerStarted","Data":"40d89d1fc5be6477b8c53fbfb348e6724b876c796ea2be99fdc24417b78e158f"}
Oct 30 21:43:00.768379 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.768076    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:00.771190 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.771165    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"88c16172ea79e41bf7c7bd6a8417a7847f7251a1779e3f0738c13e3f237488bb"}
Oct 30 21:43:00.771333 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.771227    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"5cf0159dc408a70d4e41cc72c48b150289325f0790b43cd3c543d24a05604d9e"}
Oct 30 21:43:00.771333 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.771275    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:00.771333 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.771313    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" event={"ID":"a9185dde-8443-434d-9df7-27d7e287b372","Type":"ContainerStarted","Data":"be83faf8ba0a349fb2bf40ff6dfe28e44afdd0ab97c7f3a729193f0cbcec4eab"}
Oct 30 21:43:00.773329 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.773302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:43:00.773443 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.773342    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb" event={"ID":"e8b30f97-b15d-4c82-b314-2849193c3a98","Type":"ContainerStarted","Data":"93dd8fc9ec89c29737b13c667f0c50a93fc2fc3b3ad6ed4bb29d0e7d74d85bbe"}
Oct 30 21:43:00.773580 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.773565    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:00.878557 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:00.878494    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz" podStartSLOduration=1.9788228559999999 podStartE2EDuration="3.878476405s" totalImagesPullingTime="1.899653549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:00.877846144 +0000 UTC m=+63.096013967" watchObservedRunningTime="2026-10-30 21:43:00.878476405 +0000 UTC m=+63.096644232"
Oct 30 21:43:01.219143 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:01.219089    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4" podStartSLOduration=2.272931555 podStartE2EDuration="6.219075296s" totalImagesPullingTime="3.946143741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:01.218668866 +0000 UTC m=+63.436836700" watchObservedRunningTime="2026-10-30 21:43:01.219075296 +0000 UTC m=+63.437243119"
Oct 30 21:43:01.247368 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:01.247332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:01.414116 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:01.414085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:02.058982 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:02.058949    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:02.216583 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:02.216548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:02.376799 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:02.376686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.053682 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.053647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.107267 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.107235    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:03.110814 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.110798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.161447 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.161390    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:03.184113 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184278 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184446 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184446 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184446 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184530 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184530 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184530 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184617 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.184617 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.184585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.207659 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207627    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:43:03.207851 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207627    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:43:03.207851 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207656    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:43:03.207851 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207690    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:43:03.207851 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207710    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:43:03.207851 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207729    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:43:03.208032 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207866    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:43:03.208032 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.207902    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:43:03.255889 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.255854    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89d9x\""
Oct 30 21:43:03.257282 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.257260    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:43:03.285707 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.285882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.285882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.285882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.285882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.285882 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.285983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.286007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.286028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.286073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286135 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.286114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.286501 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.286473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.287374 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.287346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.287475 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.287440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.288838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.288750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.288838 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.288817    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.289007 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.288909    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.289445 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.289417    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.289557 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.289449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.290321 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.290278    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.290469 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.290449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.290469 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.290450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.290886 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.290867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.328115 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.328006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.342240 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.342181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.434055 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.434024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:03.507008 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.506974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.661561 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.661486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:03.895410 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:03.895385    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:03.897257 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:43:03.897228    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700a06ab_4440_42e1_abcc_ca01af3ffd59.slice/crio-f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8 WatchSource:0}: Error finding container f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8: Status 404 returned error can't find the container with id f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8
Oct 30 21:43:04.094011 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.093972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:04.143291 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.143262    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:43:04.146358 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.146328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:04.194950 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.194897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:04.194950 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.194954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:04.245604 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.245568    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:43:04.245765 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.245707    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:43:04.258252 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.258066    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:04.305843 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.305816    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:43:04.318142 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.318110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:04.389559 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.389477    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8c5b8\""
Oct 30 21:43:04.390529 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.390499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:04.394707 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.394634    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hmtv9\""
Oct 30 21:43:04.395681 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.395667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:04.399577 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.399555    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s2r8\""
Oct 30 21:43:04.401657 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.401575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:04.764461 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.764431    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wr6rh"]
Oct 30 21:43:04.788403 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.788370    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8"}
Oct 30 21:43:04.797728 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:43:04.797705    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54348157_31ac_475e_90a1_32b963be27a3.slice/crio-cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496 WatchSource:0}: Error finding container cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496: Status 404 returned error can't find the container with id cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496
Oct 30 21:43:04.806368 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.806348    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l84ww"]
Oct 30 21:43:04.807996 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:43:04.807969    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099cdf15_a1f3_456c_bbf5_34b2363b6142.slice/crio-d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610 WatchSource:0}: Error finding container d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610: Status 404 returned error can't find the container with id d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610
Oct 30 21:43:04.809601 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:04.809578    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68rh6"]
Oct 30 21:43:04.812221 ip-10-0-131-231 kubenswrapper[2578]: W1030 21:43:04.812198    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e23aa0b_cefb_4c96_b4a6_d48235deb03b.slice/crio-333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d WatchSource:0}: Error finding container 333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d: Status 404 returned error can't find the container with id 333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d
Oct 30 21:43:05.075431 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.075350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:05.240232 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.240197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:05.792657 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.792621    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l84ww" event={"ID":"099cdf15-a1f3-456c-bbf5-34b2363b6142","Type":"ContainerStarted","Data":"d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610"}
Oct 30 21:43:05.793663 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.793637    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wr6rh" event={"ID":"54348157-31ac-475e-90a1-32b963be27a3","Type":"ContainerStarted","Data":"cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496"}
Oct 30 21:43:05.795044 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.795020    2578 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2" exitCode=0
Oct 30 21:43:05.795143 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.795103    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"}
Oct 30 21:43:05.796266 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.796248    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68rh6" event={"ID":"7e23aa0b-cefb-4c96-b4a6-d48235deb03b","Type":"ContainerStarted","Data":"333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d"}
Oct 30 21:43:05.926010 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:05.925918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:06.121294 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.121252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:06.354556 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.354527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:06.563402 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.563359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:06.788649 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.788624    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:06.820127 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.820095    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l84ww" event={"ID":"099cdf15-a1f3-456c-bbf5-34b2363b6142","Type":"ContainerStarted","Data":"85300abe0b4cfbde44adb13efd8d17c48fa53b1f46b1077e7227e6057d768c95"}
Oct 30 21:43:06.846172 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:06.846135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:07.043969 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:07.043933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:07.831319 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:07.831282    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l84ww" event={"ID":"099cdf15-a1f3-456c-bbf5-34b2363b6142","Type":"ContainerStarted","Data":"d24a346422581ddd4e3c3933cad9c2274429a7a7efa4d898a4c511e29adf5c40"}
Oct 30 21:43:07.947029 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:07.946973    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l84ww" podStartSLOduration=67.099100183 podStartE2EDuration="1m8.946955745s" totalImagesPullingTime="1.847855562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:07.945902312 +0000 UTC m=+70.164070134" watchObservedRunningTime="2026-10-30 21:43:07.946955745 +0000 UTC m=+70.165123578"
Oct 30 21:43:08.067373 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.067335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:08.272033 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.272001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:08.839324 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839275    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"}
Oct 30 21:43:08.839803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839355    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"}
Oct 30 21:43:08.839803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839404    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"}
Oct 30 21:43:08.839803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839442    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"}
Oct 30 21:43:08.839803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839475    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"}
Oct 30 21:43:08.839803 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.839511    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerStarted","Data":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"}
Oct 30 21:43:08.964116 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:08.964055    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9829755869999999 podStartE2EDuration="5.964037673s" totalImagesPullingTime="3.969037575s" totalInitContainerRuntime="12.024511ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:08.9628423 +0000 UTC m=+71.181010122" watchObservedRunningTime="2026-10-30 21:43:08.964037673 +0000 UTC m=+71.182205494"
Oct 30 21:43:09.228326 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.228293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:09.391697 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.391620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:09.656734 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.656659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:09.844871 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.844825    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68rh6" event={"ID":"7e23aa0b-cefb-4c96-b4a6-d48235deb03b","Type":"ContainerStarted","Data":"2428ddd0aec7c5d49b7e994fc4ccf5ed25ef79baf9c07273d0879ff1ff033fd0"}
Oct 30 21:43:09.845340 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.845273    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:09.866408 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.866376    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:09.963685 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:09.963634    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-68rh6" podStartSLOduration=66.71269729 podStartE2EDuration="1m10.963618945s" totalImagesPullingTime="4.250921655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.96254107 +0000 UTC m=+72.180708893" watchObservedRunningTime="2026-10-30 21:43:09.963618945 +0000 UTC m=+72.181786766"
Oct 30 21:43:10.850189 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:10.850152    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wr6rh" event={"ID":"54348157-31ac-475e-90a1-32b963be27a3","Type":"ContainerStarted","Data":"78722fc17486655273f1df67bec0a9dad671fb850bfe136de3b4c39362540fef"}
Oct 30 21:43:10.958020 ip-10-0-131-231 kubenswrapper[2578]: I1030 21:43:10.957953    2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:43:10.958134 ip-10-0-131-231 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:43:10.966244 ip-10-0-131-231 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:43:10.966408 ip-10-0-131-231 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:43:10.966457 ip-10-0-131-231 systemd[1]: kubelet.service: Consumed 4.079s CPU time.
Oct 30 21:43:10.985572 ip-10-0-131-231 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:43:11.043545 ip-10-0-131-231 kubenswrapper[5510]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:43:11.043545 ip-10-0-131-231 kubenswrapper[5510]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:43:11.043545 ip-10-0-131-231 kubenswrapper[5510]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:43:11.043545 ip-10-0-131-231 kubenswrapper[5510]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:43:11.046887 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046799    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.046887 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046883    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.046887 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046887    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.046887 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046891    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.046887 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046894    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046897    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046901    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046904    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046907    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046910    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046913    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046916    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046919    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046921    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046924    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046928    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046930    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046933    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046936    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046938    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046941    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046944    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046947    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.047085 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046949    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046952    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046954    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046957    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046959    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046962    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046965    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046969    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046976    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046979    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046981    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046984    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046987    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046990    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046993    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046996    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.046999    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047002    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047004    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.047556 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047008    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047011    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047014    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047016    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047019    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047021    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047024    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047027    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047029    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047032    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047036    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047039    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047041    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047045    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047050    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047053    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047056    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047059    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047061    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047064    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.048035 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047067    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047069    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047072    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047074    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047077    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047080    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047083    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047086    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047088    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047091    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047094    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047097    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047100    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047102    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047105    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047107    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047110    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047113    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047116    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.048511 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047118    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047121    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047125    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047128    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047131    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047134    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047137    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047140    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047142    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047145    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047147    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047150    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047153    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047155    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047158    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047160    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047163    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047165    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047168    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047172    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.049000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047175    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047177    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047180    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047183    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047185    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047950    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047957    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047960    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047963    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047966    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047969    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047972    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047975    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047978    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047981    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047986    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047990    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047994    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.047997    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048000    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.049489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048003    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048005    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048008    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048010    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048013    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048016    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048018    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048021    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048024    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048026    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048029    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048032    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048035    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048038    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048041    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048043    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048046    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048049    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048052    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048055    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.050029 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048058    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048060    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048064    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048067    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048070    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048072    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048075    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048078    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048081    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048083    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048086    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048089    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048091    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048094    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048097    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048099    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048102    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048105    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048108    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.050517 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048111    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048114    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048117    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048120    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048122    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048126    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048130    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048134    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048136    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048139    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048141    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048144    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048147    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048150    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048153    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048155    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048158    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048161    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.051000 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048163    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048166    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048168    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048171    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048175    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048177    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048180    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048182    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048185    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048188    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048190    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048193    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048195    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048198    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048201    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048204    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048206    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048209    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048211    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048214    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.051496 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048216    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048219    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048221    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048224    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048227    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048231    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048234    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048236    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048239    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048241    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048244    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048247    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048249    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048252    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048747    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048753    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048756    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048759    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048762    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048764    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.051986 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048767    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048770    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048772    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048775    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048800    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048805    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048810    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048813    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048816    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048818    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048821    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048824    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048827    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048830    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048832    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048835    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048838    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048841    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048844    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048847    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.052494 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048850    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048852    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048855    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048858    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048860    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048863    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048865    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048868    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048871    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048874    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048877    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048880    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048882    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048885    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048888    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048891    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048893    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048896    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048899    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048902    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.053008 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048904    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048907    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048909    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048912    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048914    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048917    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048920    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048922    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048925    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048928    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048931    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048933    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048936    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048939    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048941    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048944    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048947    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048949    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048952    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.053482 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048954    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048957    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048959    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048962    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048965    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048968    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048970    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048973    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048975    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048978    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048980    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048983    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048985    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048988    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048991    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048993    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048996    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.048998    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049001    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.053953 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049004    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049007    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049009    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049012    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049015    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049018    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049021    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049026    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049030    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049033    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049036    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049039    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049043    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049046    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049049    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049052    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049054    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049058    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049061    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.054425 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049064    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049066    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049069    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049165    5510 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049173    5510 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049179    5510 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049184    5510 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049188    5510 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049192    5510 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049196    5510 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049200    5510 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049203    5510 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049206    5510 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049211    5510 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049214    5510 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049218    5510 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049221    5510 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049224    5510 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049227    5510 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049230    5510 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049233    5510 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049264    5510 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049268    5510 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:43:11.054902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049271    5510 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049275    5510 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049278    5510 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049283    5510 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049286    5510 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049289    5510 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049292    5510 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049295    5510 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049299    5510 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049302    5510 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049309    5510 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049312    5510 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049316    5510 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049320    5510 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049323    5510 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049326    5510 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049329    5510 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049332    5510 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049338    5510 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049341    5510 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049344    5510 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049347    5510 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049352    5510 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049360    5510 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:43:11.055629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049363    5510 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049367    5510 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049370    5510 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049373    5510 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049377    5510 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049380    5510 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049383    5510 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049385    5510 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049388    5510 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049391    5510 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049395    5510 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049398    5510 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049401    5510 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049405    5510 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049408    5510 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049411    5510 flags.go:64] FLAG: --help="false"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049414    5510 flags.go:64] FLAG: --hostname-override="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049418    5510 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049423    5510 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049426    5510 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049429    5510 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049433    5510 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049436    5510 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:43:11.056282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049439    5510 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049442    5510 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049445    5510 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049448    5510 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049451    5510 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049454    5510 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049457    5510 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049461    5510 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049464    5510 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049467    5510 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049470    5510 flags.go:64] FLAG: --lock-file=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049473    5510 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049476    5510 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049480    5510 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049485    5510 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049488    5510 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049491    5510 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049494    5510 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049497    5510 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049501    5510 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049504    5510 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049507    5510 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049511    5510 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049514    5510 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049518    5510 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:43:11.056890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049522    5510 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049525    5510 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049530    5510 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049533    5510 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049536    5510 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049539    5510 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049542    5510 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049554    5510 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049557    5510 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049560    5510 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049563    5510 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049566    5510 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049569    5510 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049572    5510 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049576    5510 flags.go:64] FLAG: --port="10250"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049580    5510 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049583    5510 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ef5bfe7fe60adc8b"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049586    5510 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049589    5510 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049592    5510 flags.go:64] FLAG: --register-node="true"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049595    5510 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049599    5510 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049602    5510 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049605    5510 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049608    5510 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049611    5510 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:43:11.057503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049614    5510 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049618    5510 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049621    5510 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049624    5510 flags.go:64] FLAG: --runonce="false"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049627    5510 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049630    5510 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049632    5510 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049635    5510 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049640    5510 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049647    5510 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049650    5510 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049653    5510 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049656    5510 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049659    5510 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049662    5510 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049665    5510 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049672    5510 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049675    5510 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049678    5510 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049684    5510 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049689    5510 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049692    5510 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049695    5510 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049698    5510 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049701    5510 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:43:11.058160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049705    5510 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049708    5510 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049711    5510 flags.go:64] FLAG: --v="2"
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049715    5510 flags.go:64] FLAG: --version="false"
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049719    5510 flags.go:64] FLAG: --vmodule=""
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049723    5510 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.049727    5510 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049877    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049882    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049886    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049889    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049892    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049895    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049898    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049900    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049905    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049908    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049911    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049914    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049917    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049920    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.059079 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049922    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049925    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049928    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049931    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049933    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049936    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049940    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049943    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049946    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049949    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049951    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049954    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049957    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049967    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049970    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049975    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049979    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049982    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049985    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.059852 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049987    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049990    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049993    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049995    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.049998    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050001    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050003    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050008    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050010    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050013    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050017    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050020    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050022    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050025    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050027    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050030    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050034    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050036    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050039    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.060561 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050043    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050046    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050048    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050051    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050054    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050056    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050059    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050063    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050066    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050069    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050071    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050074    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050077    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050079    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050082    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050085    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050087    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050090    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050093    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050096    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.061402 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050100    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050103    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050106    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050109    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050112    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050114    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050117    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050120    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050122    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050125    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050128    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050130    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050134    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050137    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050140    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050143    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050145    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050148    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050151    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.061901 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050153    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050156    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050158    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050161    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050164    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050166    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050169    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050172    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050174    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050177    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050179    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050182    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050185    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050189    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050192    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.050217    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:43:11.062387 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050255    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050259    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050263    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050266    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050269    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050272    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050274    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050277    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050280    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050284    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050287    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050290    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050292    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050295    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050298    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050300    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050303    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050306    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050310    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.062860 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050314    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050318    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050321    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050324    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050326    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050329    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050332    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050335    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050337    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050340    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050343    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050347    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050349    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050352    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050355    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050357    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050360    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050363    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050366    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.063438 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050369    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050371    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050374    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050378    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050381    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050383    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050386    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050389    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050392    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050394    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050398    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050401    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050404    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050407    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050410    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050413    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050415    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050418    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050421    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050423    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.064081 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050426    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050429    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050432    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050435    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050439    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050442    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050444    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050447    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050450    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050452    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050456    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050458    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050461    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050464    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050467    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050471    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050473    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050476    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050479    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.064608 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050482    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050484    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050487    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050489    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050492    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050495    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050498    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050500    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050503    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050505    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050508    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050511    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050513    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050516    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050519    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050522    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050525    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050529    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050532    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050535    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.065097 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050538    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050540    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050543    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050545    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050549    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050552    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050555    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050557    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.050562    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.054106    5510 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.054120    5510 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.054138    5510 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.054142    5510 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054195    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054201    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054205    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054208    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054211    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054214    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.065566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054217    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054221    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054223    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054226    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054229    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054232    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054235    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054237    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054240    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054243    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054246    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054249    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054252    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054255    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054258    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054260    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054263    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054267    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054270    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.066051 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054272    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054275    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054277    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054280    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054283    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054285    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054288    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054290    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054293    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054296    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054298    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054301    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054304    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054307    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054309    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054312    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054316    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054320    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054323    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054325    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.066501 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054329    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054332    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054376    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054379    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054383    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054386    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054389    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054392    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054394    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054397    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054400    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054408    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054411    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054414    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054417    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054421    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054426    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054429    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054432    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.066981 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054435    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054438    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054441    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054443    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054446    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054449    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054452    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054454    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054457    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054459    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054462    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054465    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054467    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054470    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054472    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054476    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054479    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054481    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054484    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054487    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.067430 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054489    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054492    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054495    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054498    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054500    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054503    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054506    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054509    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054511    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054514    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054516    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054519    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054522    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054524    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054527    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054530    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054532    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054535    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054537    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.067997 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054540    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054543    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054546    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.054572    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054705    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054709    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054712    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054716    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054719    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054721    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054724    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054726    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054729    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054732    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054734    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054737    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.068440 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054740    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054742    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054745    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054748    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054751    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054754    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054756    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054759    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054762    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054764    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054767    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054770    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054772    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054775    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054795    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054798    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054801    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054804    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054807    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054809    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.068872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054812    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054815    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054818    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054821    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054823    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054826    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054829    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054831    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054834    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054836    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054839    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054842    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054845    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054849    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054852    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054856    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054859    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054862    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054866    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.069476 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054870    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054872    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054875    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054877    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054880    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054883    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054885    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054888    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054890    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054893    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054895    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054898    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054901    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054904    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054907    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054910    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054913    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054916    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054918    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.070017 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054921    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054924    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054926    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054929    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054931    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054934    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054936    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054939    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054942    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054945    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054948    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054951    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054953    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054956    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054959    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054961    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054964    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054967    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054969    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.070616 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054972    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054974    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054977    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054980    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054982    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054985    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054987    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054991    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054993    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054996    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.054999    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055001    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055008    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055011    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055014    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055017    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.071307 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055019    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.055066    5510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055098    5510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055103    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055106    5510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055109    5510 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055112    5510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055115    5510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055118    5510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055121    5510 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055124    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055127    5510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055129    5510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055132    5510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055135    5510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055137    5510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:43:11.071707 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055140    5510 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055142    5510 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055145    5510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055147    5510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055150    5510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055153    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055155    5510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055158    5510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055161    5510 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055163    5510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055166    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055169    5510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055171    5510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055174    5510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055177    5510 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055179    5510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055182    5510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055185    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055187    5510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055190    5510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:43:11.072150 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055192    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055195    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055197    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055200    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055202    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055205    5510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055209    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055212    5510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055214    5510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055217    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055220    5510 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055222    5510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055226    5510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055230    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055233    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055235    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055240    5510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055244    5510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055250    5510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:43:11.072627 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055253    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055256    5510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055258    5510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055276    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055279    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055282    5510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055285    5510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055288    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055290    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055293    5510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055295    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055298    5510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055301    5510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055303    5510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055306    5510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055309    5510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055312    5510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055314    5510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055317    5510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055320    5510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:43:11.073158 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055323    5510 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055326    5510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055328    5510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055331    5510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055334    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055336    5510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055339    5510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055341    5510 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055344    5510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055347    5510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055349    5510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055354    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055357    5510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055359    5510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055362    5510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055365    5510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055368    5510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055370    5510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055373    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:43:11.073649 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055376    5510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055378    5510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055381    5510 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055383    5510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055386    5510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055388    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055391    5510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055394    5510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055397    5510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055399    5510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055402    5510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055404    5510 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055407    5510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:11.055411    5510 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.055465    5510 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056520    5510 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056575    5510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056760    5510 server.go:1014] "Starting client certificate rotation"
Oct 30 21:43:11.074379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056895    5510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056954    5510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:58 +0000 UTC" deadline="2028-06-11 17:45:24.41651849 +0000 UTC"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.056980    5510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14156h2m13.359540767s"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.057213    5510 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.058224    5510 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.059968    5510 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.062507    5510 log.go:25] "Validated CRI v1 image API"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.063239    5510 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:43:11.075025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.065192    5510 fs.go:135] Filesystem UUIDs: map[120aae6e-3ed4-4ce9-bb93-092fea47ebc7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dc0b2a7b-df43-47cd-8cf0-005c4095efb3:/dev/nvme0n1p3]
Oct 30 21:43:11.075268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.065207    5510 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79516c8d3e2dff938854e1ffa92ef32523a106a5ee20984e522dd1f544b7005d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79516c8d3e2dff938854e1ffa92ef32523a106a5ee20984e522dd1f544b7005d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39/userdata/shm major:0 minor:282 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~projected/kube-api-access-jml4r:{mountpoint:/var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~projected/kube-api-access-jml4r major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~secret/metrics-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1012b47f-8e91-41ca-bc3e-8597142bce89/volumes/kubernetes.io~projected/kube-api-access-lshbk:{mountpoint:/var/lib/kubelet/pods/1012b47f-8e91-41ca-bc3e-8597142bce89/volumes/kubernetes.io~projected/kube-api-access-lshbk major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~projected/kube-api-access-fv4nt:{mountpoint:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~projected/kube-api-access-fv4nt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~projected/kube-api-access-fq4s5:{mountpoint:/var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~projected/kube-api-access-fq4s5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54348157-31ac-475e-90a1-32b963be27a3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/54348157-31ac-475e-90a1-32b963be27a3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bf6b327-54b3-42dd-bf21-4c4ecd526af7/volumes/kubernetes.io~projected/kube-api-access-twznq:{mountpoint:/var/lib/kubelet/pods/5bf6b327-54b3-42dd-bf21-4c4ecd526af7/volumes/kubernetes.io~projected/kube-api-access-twznq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67a0fce0-c116-4eb1-b233-066ba56adb42/volumes/kubernetes.io~projected/kube-api-access-fq7cq:{mountpoint:/var/lib/kubelet/pods/67a0fce0-c116-4eb1-b233-066ba56adb42/volumes/kubernetes.io~projected/kube-api-access-fq7cq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ef60f7-bf87-4e70-bf23-4671a0ea4f23/volumes/kubernetes.io~projected/kube-api-access-sd5kf:{mountpoint:/var/lib/kubelet/pods/69ef60f7-bf87-4e70-bf23-4671a0ea4f23/volumes/kubernetes.io~projected/kube-api-access-sd5kf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/alertmanager/9 major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/config-reloader/6 major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~empty-dir/config-out major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/kube-api-access-s4tlr:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/kube-api-access-s4tlr major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/tls-assets major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/config-volume major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/web-config major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~projected/kube-api-access-wxd48:{mountpoint:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~projected/kube-api-access-wxd48 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e23aa0b-cefb-4c96-b4a6-d48235deb03b/volumes/kubernetes.io~projected/kube-api-access-4dslv:{mountpoint:/var/lib/kubelet/pods/7e23aa0b-cefb-4c96-b4a6-d48235deb03b/volumes/kubernetes.io~projected/kube-api-access-4dslv major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~projected/kube-api-access-z2lqd:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~projected/kube-api-access-z2lqd major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~projected/kube-api-access-bx9c2:{mountpoint:/var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~projected/kube-api-access-bx9c2 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b640ceb8-5397-4ff2-9428-ab03c9cf3474/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b640ceb8-5397-4ff2-9428-ab03c9cf3474/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~projected/kube-api-access-8z99j:{mountpoint:/var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~projected/kube-api-access-8z99j major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~projected/kube-api-access-8tnxl:{mountpoint:/var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~projected/kube-api-access-8tnxl major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7b36d9e-7722-4db1-acb4-7a75f0df705c/volumes/kubernetes.io~projected/kube-api-access-ltqr2:{mountpoint:/var/lib/kubelet/pods/d7b36d9e-7722-4db1-acb4-7a75f0df705c/volumes/kubernetes.io~projected/kube-api-access-ltqr2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~projected/kube-api-access-887zx:{mountpoint:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~projected/kube-api-access-887zx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~projected/kube-api-access-dj992:{mountpoint:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~projected/kube-api-access-dj992 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8c9495d-4278-42d4-98dc-a2e8b1dc356d/volumes/kubernetes.io~projected/kube-api-access-4mffs:{mountpoint:/var/lib/kubelet/pods/f8c9495d-4278-42d4-98dc-a2e8b1dc356d/volumes/kubernetes.io~projected/kube-api-access-4mffs major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5298e84fed6c93c8b88ea245158aa83385b273548c6d2dad39321f15e3e023d0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8119c9698cfbea22659b05d93ba0d9502346a79ea7adb52e877f1eca243be3bc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5194ef22ed23c7c6d7cf108ebc8a6ac7635f92ec2677c25737559422a75ac4de/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e08f397ab402c724c12516fe1eb9d7c2403f527c7fa1d1fdbee32f4f919412f5/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8fdcafa341b1366134a1e6f86c2da6077743025a2b96239a550fe4f0d509881c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2f0d772eb5514d48c1c96f0f79fd0b07f58e50ed51fea9452a791498ad15c4eb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b3f2742961f023b3bb2510368b469298ba5121aa9f9326fcd9585e3dd14fcee1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e5e56864967e9f06e2286b580fce50475afb9f737f32395fa98060d074c25d1d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b362fa441c7c6d775475767aed12bf4981656321b60c2cddb33fe644f1c2da95/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7dcabbbeef2220be05f2d4c78c9bf56098baf6eddefddf5a6f4209b1100dc908/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f77c8183fc4064c5b6aa4d3874c4877010a01d905631416b0f884f64a924ab3e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4308313f3abc1e05fbe8e067eff1df3917c7961060045207ba9d392985d0c376/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1ef3dcfe4bb40fd2318d624b09b731a1e3d735c71abcccf38e7e1c5ee7fe1b51/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/030137da47a015671bc5600858a478127e4f66536d0cd3e3f9d5ccc21db38af8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/06a17b9308764731316cc80d23e7b2d7125bad16be464553ddf3444a128db25f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/04c697c0733e5aafb585692306b65d7d562f152ca05dd7b0d44ab3e5454adaf9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/144a8cfcb2fd8c4007e6b71b7638adac268ab0022aee205c77c4632a5c7ef8a0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7d8e28e07d8a188f687af450991c4eff2380929546c15e1a0d5fed1cf3f5e156/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c7f57b3151eb41117fba9a02f2b873fa90ed2f714488621a43b35f7071bc90f1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/36870c17e1b96e67990122675e72bb042ef6993b2ad1aea21fcecaddad9b187c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/53d8f3ab7e2923d641014e9447badb345e5ae20b406136b3649a8ee53b682245/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7aad84c1cd0116d4bf8d632832713a94d5611964bb09f6a11c1039e5d2591cdc/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/95276776c397c73d13263de893c853e9e77032cb3ad5a3dbc19cc473b8a98c87/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6b39bb1ede768d61f3c8e257287bb8cdc98f5259cfe397d770f06e4d1d27bc24/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/517582967f4fceedab87ddf3301fd175c909c9f974721a1702faf90a0f5cc10d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/4c9b829799a8ae95b38b641df6db7f206f43667deacb88a95863153e6d7ad769/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/c372b58f90bbad33a468c619834373b136fa2722a18f5ea61594bd71ffaf2621/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/d2c13c45a0088424752f111fac682d066f7b676a74a209485532e0d216ef09f7/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/98cea876af4da709e3e1abfc28eb249b7c5f6d2fbb46e1ce76c9a05f851c2883/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/8ae0b20dd73e0fe14340db02cee6d2e15e0781497b3e263cff816af883f66491/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3d48c2eb016ffdc5d8c493010c74afb861103d767deb300089bf0350624782b4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/ea915d3187fdf767b8efcec853dc8675addda3fad883fc45303e2e7a6466df35/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/957c5cba1a5f3a6ef806bb012ed72dc8a79445e5c4454f436647d4e71c8874e0/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/92f89059cf37c9a2540bb6f144c08eed0037165d4e113acbd7529eca658325e4/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d2a9389406cf7c779d8b34fef36bb563ea4c721fc16885685061ca3cd57a16d6/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/00e090f080ab69874a658d8f2f4819ac4252451701e9b67215d8d721e72e52e2/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c7ed9610308b7fca2573f2848dd7ce92ef39cccc8c0a2f6e07647978a903be8a/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/1a4255bc4f36a60b1f0102d8c013faf3acf0c73d7ddbcc22808a6b8fa72d0afe/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/7c6daac2aaf941de904e0bc3ff740f5aed7875ba81bb4b92c566b4aa1c1cd41f/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d6be4d49f083c51c83ef9475734c101bfef62b4d680322b210500a080b917805/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c62a683dec3338b3abc4e759485ee8dbce41a11dd5a68cb0a4b95c980e9eed73/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/32c50a8849451bc18e04923f2a704fc0086c28bedcc3a152750627e2e85d6ad3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/4666859ed2468c6ec69d1d376ae90fbf8d43a5312d43e1f71500a23f4f1f7d7a/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/64b3b2c77f8b8450694178374e1168207722e8e551dfdc01b101eb0b2bf67fb7/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/6ac744ae5c2fe6a0ef319ef170e9e373a69df042c4e3b9947efa31c4e01a9f9a/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/7aa3241f6f37aa46ab66a9169ca780787648a3c0ccd5f570bb2aa6ee27f517f7/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/e6245919e8fcef46d5b44a1ed5e9f56a642ae2f28a2475e47c0e00c6c101c354/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/3e36e0aa598444f6fca38a35240636a85f311164ee749534cf9ce9b375238a5d/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d970f9dd83e1a69860dec1564f86194eef3618cfed086fae7f26f2403a8e7214/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/4dc87795bab4107f8ec89bef3733e01b7737e0d0227c5e7c23ac6f6277818804/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/ae33c95168800f1b0e44576c6b7408c4048ba372ca888acef558c95e813c055c/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/b93740ac1f59c95b0e97cbed4020e93c65361da742127c45b3685549b7182547/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/eeb0850ebb6f15dae9c0cd89220a08ae986688bc63c3422fb93a876e824b5416/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/350a16ec766c6a38604fcf8d64e4941f56d12719a9c65be086348502e2fe0a28/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/25046fbea818775de3af4f6530320edb3e207e7efe5530efc6331b378420f354/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/cb0200a71dc30ff6477d2d0a013ec99e0f2de4afe9095f5f12f45d9d57340b10/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/0033b09c6a3dd33e7ddd44d0964cefe2fb04619485411634479b5920e2a953de/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/97903d2b20b61c306d6ab03543fd27d67ad860ce40aedf4a205d16c20a6eae8e/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/40d01ba8f89d2678d3e4ad1ec3ee7acf783a5786a3a4087e0f8677823f698a99/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/d7040d325fe253e20cd6fc625030144a0ece15860cad4591c0e8e408416341be/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e6630a937a6af128a4b2f2bc789c638fead02232d348ec63ffc00aaef9985cd8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/acd69aa7f8ada9bf11a0472a95412333bdcee170116f74039657d4066caee81e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/adee0cd570fc4c654cb4a35dc4722d359902d2f74acf9f54aacf6f5cb57722da/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/154e710749ddd7ec79095b31ff486b10732a8c62e30e9cec902f21fcfbdcba38/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c88cde9e68d696de0a01565c21a714c6e9f70b7f70eaa59d1f4a1dc65c3ce005/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/87eb504164cc9b3e8c6508e9c488f03219c142655dce09b1ada40e96c44d14ea/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/17a1e6a5803c81a8730c754d4ef68413ba73ef899fd70b55fd63e7498b7a2da8/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/669da0beadeb66b044d6347b721a512dae7722f4ef917a8e368183cd759fd2e2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/5175f53694957f22470b2d241438f5baa7cc2bce25bef513affa25b3713de68e/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.072480    5510 manager.go:217] Machine: {Timestamp:2026-10-30 21:43:11.072013247 +0000 UTC m=+0.055920119 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106203 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb14ea987b096a0500ae82b33aaa9 SystemUUID:ec2bb14e-a987-b096-a050-0ae82b33aaa9 BootID:34068830-3797-4b63-9fbf-4fd55ac4060d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5bf6b327-54b3-42dd-bf21-4c4ecd526af7/volumes/kubernetes.io~projected/kube-api-access-twznq DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7888854c94e070bf0dbd4e47fee544722ffd8a0bcbcd8e27a61554d659cd41b6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~projected/kube-api-access-8tnxl DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~projected/kube-api-access-bx9c2 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~projected/kube-api-access-wxd48 DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21644eb8ef503e26f4b6f687b49aa5722bc64ec9ec7673e5b091a643a0d3e466/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03f61de11906709a0cde9c9417a0427e80b1e7a58d47cbb418f4dd908034d647/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b640ceb8-5397-4ff2-9428-ab03c9cf3474/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67a0fce0-c116-4eb1-b233-066ba56adb42/volumes/kubernetes.io~projected/kube-api-access-fq7cq DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b373bccf41df2bf2204722849ec9736c349a08f715920c551b41718e980d934/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e23aa0b-cefb-4c96-b4a6-d48235deb03b/volumes/kubernetes.io~projected/kube-api-access-4dslv DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79516c8d3e2dff938854e1ffa92ef32523a106a5ee20984e522dd1f544b7005d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41b907cc79f0c4d6a4cebaad7b7a3b99cb05d3b17fe0c016d2e5570eb14d900a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1012b47f-8e91-41ca-bc3e-8597142bce89/volumes/kubernetes.io~projected/kube-api-access-lshbk DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/465b6111-ca93-426c-9901-e452be8ba99e/volumes/kubernetes.io~projected/kube-api-access-fq4s5 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a69c6469058d6b985b78b769c842a030b3af6b7a3fad5400c4057ed8340334ea/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a8572a-f8ad-4c98-8758-cc25c4823aaa/volumes/kubernetes.io~projected/kube-api-access-8z99j DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~projected/kube-api-access-887zx DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:342 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d02679970d90794a9c4ee5e835bbf38027e843b8ab8d142f3d7da2de4fff1610/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37a31c33e625c10736e483069e3897f0fa2479b00757ce7fd4f58796a73270cd/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~projected/kube-api-access-jml4r DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/099cdf15-a1f3-456c-bbf5-34b2363b6142/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54348157-31ac-475e-90a1-32b963be27a3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb36ba154f0c1e51f1da02ad20c3a453c7ae619db44c8c8f640e97cd4ed90496/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d877ceb525e5be48bd695b1f018d7ff2d98e692e5404366b68351fbedb49d3bd/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa23b55069630dea527bb3c5c5ee08544ec0fa9b4902dca817da53e1b557233c/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70c2c9367b5e8970ef022f1da7d2645fbdf0a0cc9714e844337d6322ccf01c2d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5f04488495b38752586a0a15620dcb599b6819a1d4fa6123e05447a6ed2e5aa/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/740d086461516b6f762df12a61fd3650973fb3c1100f27d28934e8f0bbb67f9b/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76a1da5e-7477-453d-bd89-cbe7f9e7b20a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4fa604997eb0384974eed54f89be6b79d06f3a92cef76ac021264711edbbe39/userdata/shm DeviceMajor:0 DeviceMinor:282 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f8c9495d-4278-42d4-98dc-a2e8b1dc356d/volumes/kubernetes.io~projected/kube-api-access-4mffs DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3e405b0841c9cf3da26f27ce52b67e33a4b0d9cbb0a308a05daf5be0f6dabd4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/333031dde25d6783d04f120bc1b0b67e9eb10813b87cbb3472b0bbbec3ad1a7d/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c08463d53573f915c02e88b2a16d503b855a1aaef687ee853538f0aac8b211f5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf6dc066-7ed1-4e73-86ad-5455718c1106/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~projected/kube-api-access-z2lqd DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d7b36d9e-7722-4db1-acb4-7a75f0df705c/volumes/kubernetes.io~projected/kube-api-access-ltqr2 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b30f97-b15d-4c82-b314-2849193c3a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c728bb-8670-4f08-a095-b0b9aa9adcc7/volumes/kubernetes.io~projected/kube-api-access-dj992 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab8d50c8-864a-4758-9ece-62c693d9cffd/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes/kubernetes.io~projected/kube-api-access-s4tlr DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3caeefbd-7ff9-4082-837d-48155be1d7b5/volumes/kubernetes.io~projected/kube-api-access-fv4nt DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9185dde-8443-434d-9df7-27d7e287b372/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69ef60f7-bf87-4e70-bf23-4671a0ea4f23/volumes/kubernetes.io~projected/kube-api-access-sd5kf DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:333031dde25d678 MacAddress:22:32:e2:7f:17:dc Speed:10000 Mtu:8901} {Name:37a31c33e625c10 MacAddress:72:db:7a:31:aa:ef Speed:10000 Mtu:8901} {Name:70c2c9367b5e897 MacAddress:26:03:b1:55:ef:1b Speed:10000 Mtu:8901} {Name:7888854c94e070b MacAddress:e6:a4:29:f5:10:c8 Speed:10000 Mtu:8901} {Name:a69c6469058d6b9 MacAddress:72:b6:46:18:56:6f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fc:01:46:19 Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:45:84:d8:1d:7a Speed:0 Mtu:8901} {Name:c4fa604997eb038 MacAddress:7a:f2:5e:62:b1:f9 Speed:10000 Mtu:8901} {Name:cb36ba154f0c1e5 MacAddress:ae:d6:dc:c8:e6:cf Speed:10000 Mtu:8901} {Name:d02679970d90794 MacAddress:5a:ca:e0:6e:68:d3 Speed:10000 Mtu:8901} {Name:d877ceb525e5be4 MacAddress:4e:b5:c7:33:99:84 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fc:01:46:19 Speed:0 Mtu:9001} {Name:f7e1979c3914f58 MacAddress:9e:5b:33:22:62:ef Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:16:5a:36:03:41:b5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:c7:2e:37:2e:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.072858    5510 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.072901    5510 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073172    5510 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073194    5510 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-231.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073313    5510 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073320    5510 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073330    5510 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073349    5510 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073452    5510 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073606    5510 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073718    5510 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073754    5510 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073768    5510 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073793    5510 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073806    5510 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.073817    5510 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.074471    5510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.074480    5510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:43:11.075503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.074882    5510 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:43:11.077232 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077218    5510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:43:11.077232 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077232    5510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:43:11.077327 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077318    5510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:43:11.077509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077499    5510 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:43:11.077538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077528    5510 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:43:11.077618 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077610    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:43:11.077647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077623    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:43:11.077647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077630    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:43:11.077647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077635    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:43:11.077647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077640    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:43:11.077647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077646    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077652    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077659    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077665    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077671    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077685    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077694    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077718    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:43:11.077797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.077723    5510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:43:11.081043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.081030    5510 server.go:1263] "Started kubelet"
Oct 30 21:43:11.081883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.081104    5510 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:43:11.082096 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.081984    5510 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:43:11.082096 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.082070    5510 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:43:11.083473 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.082459    5510 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:43:11.083594 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.083575    5510 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:43:11.085922 ip-10-0-131-231 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087000    5510 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087025    5510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087103    5510 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087116    5510 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087125    5510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:37:00 +0000 UTC" deadline="2028-07-12 11:41:02.05363214 +0000 UTC"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087147    5510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14893h57m50.966488716s"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.087188    5510 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087235    5510 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087683    5510 factory.go:55] Registering systemd factory
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.087697    5510 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088147    5510 factory.go:153] Registering CRI-O factory
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088165    5510 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088235    5510 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088254    5510 factory.go:103] Registering Raw factory
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088286    5510 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.088645    5510 manager.go:319] Starting recovery of all containers
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089538    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089586    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089601    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab8d50c8-864a-4758-9ece-62c693d9cffd" volumeName="kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089615    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1012b47f-8e91-41ca-bc3e-8597142bce89" volumeName="kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089631    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089646    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089657    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e23aa0b-cefb-4c96-b4a6-d48235deb03b" volumeName="kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089669    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b640ceb8-5397-4ff2-9428-ab03c9cf3474" volumeName="kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089685    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b30f97-b15d-4c82-b314-2849193c3a98" volumeName="kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089698    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089709    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="099cdf15-a1f3-456c-bbf5-34b2363b6142" volumeName="kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089724    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3caeefbd-7ff9-4082-837d-48155be1d7b5" volumeName="kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089736    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089748    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089764    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089776    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8c9495d-4278-42d4-98dc-a2e8b1dc356d" volumeName="kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089804    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089817    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" volumeName="kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089833    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ef60f7-bf87-4e70-bf23-4671a0ea4f23" volumeName="kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089846    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089859    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089871    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a8572a-f8ad-4c98-8758-cc25c4823aaa" volumeName="kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089883    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c728bb-8670-4f08-a095-b0b9aa9adcc7" volumeName="kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089896    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089910    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089922    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089968    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.089987    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf6dc066-7ed1-4e73-86ad-5455718c1106" volumeName="kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090011    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7b36d9e-7722-4db1-acb4-7a75f0df705c" volumeName="kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090026    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b30f97-b15d-4c82-b314-2849193c3a98" volumeName="kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090039    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3caeefbd-7ff9-4082-837d-48155be1d7b5" volumeName="kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090051    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090062    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="099cdf15-a1f3-456c-bbf5-34b2363b6142" volumeName="kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090073    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54348157-31ac-475e-90a1-32b963be27a3" volumeName="kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090085    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab8d50c8-864a-4758-9ece-62c693d9cffd" volumeName="kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090098    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c728bb-8670-4f08-a095-b0b9aa9adcc7" volumeName="kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090108    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c728bb-8670-4f08-a095-b0b9aa9adcc7" volumeName="kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090120    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ef60f7-bf87-4e70-bf23-4671a0ea4f23" volumeName="kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090136    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090147    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b640ceb8-5397-4ff2-9428-ab03c9cf3474" volumeName="kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090158    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b30f97-b15d-4c82-b314-2849193c3a98" volumeName="kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090173    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465b6111-ca93-426c-9901-e452be8ba99e" volumeName="kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090186    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090200    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a8572a-f8ad-4c98-8758-cc25c4823aaa" volumeName="kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090212    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090225    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1012b47f-8e91-41ca-bc3e-8597142bce89" volumeName="kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090237    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" volumeName="kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090251    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a0fce0-c116-4eb1-b233-066ba56adb42" volumeName="kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090264    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090277    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a8572a-f8ad-4c98-8758-cc25c4823aaa" volumeName="kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090290    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a8572a-f8ad-4c98-8758-cc25c4823aaa" volumeName="kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090302    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b30f97-b15d-4c82-b314-2849193c3a98" volumeName="kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090314    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" volumeName="kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090327    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" volumeName="kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090338    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090350    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7b36d9e-7722-4db1-acb4-7a75f0df705c" volumeName="kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090371    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c728bb-8670-4f08-a095-b0b9aa9adcc7" volumeName="kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090384    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3caeefbd-7ff9-4082-837d-48155be1d7b5" volumeName="kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090395    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67a0fce0-c116-4eb1-b233-066ba56adb42" volumeName="kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090407    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090419    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" volumeName="kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090430    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ef60f7-bf87-4e70-bf23-4671a0ea4f23" volumeName="kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090441    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090452    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090462    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c728bb-8670-4f08-a095-b0b9aa9adcc7" volumeName="kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090475    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465b6111-ca93-426c-9901-e452be8ba99e" volumeName="kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090486    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090496    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9185dde-8443-434d-9df7-27d7e287b372" volumeName="kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090506    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf6dc066-7ed1-4e73-86ad-5455718c1106" volumeName="kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090516    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465b6111-ca93-426c-9901-e452be8ba99e" volumeName="kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090529    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700a06ab-4440-42e1-abcc-ca01af3ffd59" volumeName="kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090540    5510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="465b6111-ca93-426c-9901-e452be8ba99e" volumeName="kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume" seLinuxMountContext=""
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090550    5510 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:43:11.094821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.090557    5510 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:43:11.113820 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.113762    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:43:11.113820 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.113765    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:43:11.115554 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.115526    5510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:43:11.117100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.116971    5510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:43:11.117100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.116994    5510 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:43:11.117100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.117024    5510 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:43:11.117100 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.117069    5510 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:43:11.131254 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.131225    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:43:11.131351 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.131267    5510 generic.go:356] "Generic (PLEG): container finished" podID="e8b30f97-b15d-4c82-b314-2849193c3a98" containerID="d4eaf1cca0152e83d61820b50e91d40b052b30a1ddc47bc60b5b642bc33d8920" exitCode=1
Oct 30 21:43:11.139184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139163    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="5cbc42e8b5d6f93ef11e7502229b46d1d1d225e415aa77f0e193ab0479a5152d" exitCode=0
Oct 30 21:43:11.139184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139182    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="4525b04a7a01f6ac9b55ee82c4e7c18e6526e0352c228f11c53bdf2a0cd2face" exitCode=0
Oct 30 21:43:11.139310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139189    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="e79536995a92969a90e26a3fdd8e67371fa5e65c11469d059a48d13e8a74443d" exitCode=0
Oct 30 21:43:11.139310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139195    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="2190693cdfdaf19eb2dc7657a1ce1e42e39e86fa02f2c06167a98aa1a6487a28" exitCode=0
Oct 30 21:43:11.139310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139200    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="c771d27955abfd0d934ed1199e0b25aea3cafec433025f85761e2b70975045e3" exitCode=0
Oct 30 21:43:11.139310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.139205    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bf6b327-54b3-42dd-bf21-4c4ecd526af7" containerID="61fba15671b56b70f82871a293ef59c91594acff794f22c9c58b581b9e141864" exitCode=0
Oct 30 21:43:11.142273 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.142253    5510 generic.go:356] "Generic (PLEG): container finished" podID="77f8ada4fd8175df9d44ca0f297db617" containerID="97db5a62918c7a9b4545420a8b9f17beb8f0ee4355a0f5657236497a08ab5a78" exitCode=0
Oct 30 21:43:11.147719 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.147695    5510 generic.go:356] "Generic (PLEG): container finished" podID="76a1da5e-7477-453d-bd89-cbe7f9e7b20a" containerID="d4738e8301ef32a26b1bdb8cf9edb76036801ee385ed3be0156b7778d91f6a25" exitCode=0
Oct 30 21:43:11.164670 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.164649    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2" exitCode=0
Oct 30 21:43:11.187278 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.187238    5510 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:43:11.192118 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.192097    5510 manager.go:324] Recovery completed
Oct 30 21:43:11.211767 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.211741    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:43:11.214545 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.214529    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:43:11.214630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.214559    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:43:11.214630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.214569    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:43:11.217041 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.216993    5510 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:43:11.217041 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217005    5510 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:43:11.217041 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217016    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:43:11.217205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217022    5510 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:43:11.217205 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.217132    5510 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:43:11.217276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217205    5510 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:43:11.217276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217215    5510 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:43:11.217329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217281    5510 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:43:11.217329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217287    5510 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:43:11.217329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217293    5510 policy_none.go:50] "Start"
Oct 30 21:43:11.217329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217305    5510 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:43:11.217329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217317    5510 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:43:11.217470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217448    5510 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:43:11.217470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217457    5510 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:43:11.217470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217463    5510 policy_none.go:44] "Start"
Oct 30 21:43:11.217939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217924    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:43:11.218015 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217955    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:43:11.218015 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.217975    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:43:11.221806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.221768    5510 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:43:11.226488 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226475    5510 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:43:11.226549 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.226498    5510 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:43:11.226549 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226507    5510 server.go:87] "Starting device plugin registration server"
Oct 30 21:43:11.226769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226758    5510 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:43:11.226826 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226769    5510 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:43:11.226874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226863    5510 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:43:11.226994 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226945    5510 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:43:11.226994 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226990    5510 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:43:11.226994 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.226998    5510 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:43:11.227167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.227042    5510 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:43:11.226979289Z","UUID":"3d4bb986-df23-447b-ac02-0ec9dff3484e","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:43:11.230885 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.230861    5510 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:43:11.230967 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.230869    5510 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:43:11.231121 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.230988    5510 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-231.us-west-2.compute.internal\" not found"
Oct 30 21:43:11.231367 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.231352    5510 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:43:11.269817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.269766    5510 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:43:11.269817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.269765    5510 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:43:11.269958 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.269915    5510 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:43:11.315221 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.315200    5510 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:43:11.327280 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.327264    5510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:43:11.328102 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.328087    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:43:11.328167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.328113    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:43:11.328167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.328123    5510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-231.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:43:11.328167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.328160    5510 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.361516 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.361497    5510 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:43:11.417836 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.417793    5510 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"]
Oct 30 21:43:11.418357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418321    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"0e90b7cbbf3b5bb50f8f3af03ed08b90","Type":"ContainerStarted","Data":"d05bf1d5dcacebcfea8b44107abdd518de7461c5fce222469dc015cd82a83463"}
Oct 30 21:43:11.418404 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418370    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"0e90b7cbbf3b5bb50f8f3af03ed08b90","Type":"ContainerStarted","Data":"973cbdeb372df85ab33b36c957fe65b1b27e2bba3359eba9a9745aa9807a08d2"}
Oct 30 21:43:11.418404 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418385    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerStarted","Data":"a16f14d5beb4c8cc445c1f560d6f920b0f3e072aab868dd8a10dd17a5821670a"}
Oct 30 21:43:11.418404 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418398    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerDied","Data":"97db5a62918c7a9b4545420a8b9f17beb8f0ee4355a0f5657236497a08ab5a78"}
Oct 30 21:43:11.418497 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418410    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal" event={"ID":"77f8ada4fd8175df9d44ca0f297db617","Type":"ContainerStarted","Data":"79516c8d3e2dff938854e1ffa92ef32523a106a5ee20984e522dd1f544b7005d"}
Oct 30 21:43:11.418612 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418562    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.418694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.418597    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.438895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.438875    5510 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.438958 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.438929    5510 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.480560 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.480512    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:43:11.480560 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.480546    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.481720 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.481696    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:43:11.481816 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:11.481733    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.571334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.571302    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.571334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.571336    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.571489 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.571460    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.671874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671848    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.671874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671876    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.672011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671893    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.672011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671938    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.672011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0e90b7cbbf3b5bb50f8f3af03ed08b90-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"0e90b7cbbf3b5bb50f8f3af03ed08b90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:11.672011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:11.671959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77f8ada4fd8175df9d44ca0f297db617-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" (UID: \"77f8ada4fd8175df9d44ca0f297db617\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:12.074011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.073974    5510 apiserver.go:52] "Watching apiserver"
Oct 30 21:43:12.181172 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.181144    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:12.181320 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.181252    5510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:12.215218 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.215191    5510 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:43:12.215749 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.215730    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-2cj8d","openshift-insights/insights-runtime-extractor-79gfb","kube-system/konnectivity-agent-9rtpn","kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal","openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4","openshift-multus/multus-7w2xx","openshift-dns/dns-default-dmnk2","openshift-monitoring/node-exporter-65jnb","openshift-network-operator/iptables-alerter-ccs7w","openshift-ovn-kubernetes/ovnkube-node-t6z7q","openshift-dns/node-resolver-4zb2n","openshift-multus/network-metrics-daemon-l84ww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-ktk5z","kube-system/global-pull-secret-syncer-wr6rh","openshift-ingress-canary/ingress-canary-r9zbm","openshift-monitoring/alertmanager-main-0","openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz","openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb","openshift-network-diagnostics/network-check-target-68rh6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7","openshift-cluster-node-tuning-operator/tuned-nhnkd"]
Oct 30 21:43:12.239110 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.239088    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:43:12.239209 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:12.239127    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:12.240177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.240159    5510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:43:12.240270 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:12.240203    5510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.us-west-2.compute.internal"
Oct 30 21:43:12.267251 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267221    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.267423 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267271    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:43:12.267423 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267385    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.267513 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267453    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.267617 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267602    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:43:12.267665 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267650    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:43:12.267665 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267650    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qk7p\""
Oct 30 21:43:12.267745 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.267682    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6kl\""
Oct 30 21:43:12.268524 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.268509    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwlkq\""
Oct 30 21:43:12.268721 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.268705    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.270695 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.270677    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.271045 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.271032    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:43:12.271090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.271073    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48hj5\""
Oct 30 21:43:12.271148 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.271136    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-58hb8\""
Oct 30 21:43:12.276066 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276045    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.276459 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276438    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276499    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276522    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276526    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276542    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276555    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276587    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276593    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.276631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276570    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276668    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276698    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276747    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276747    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276769    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276796    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276802    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276789    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276850    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276885    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j86j7\""
Oct 30 21:43:12.276907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.276893    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:43:12.277445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277432    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hmtv9\""
Oct 30 21:43:12.277483 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277436    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.277568 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277555    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ssgdc\""
Oct 30 21:43:12.277851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277804    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.277851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277842    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvvdp\""
Oct 30 21:43:12.277920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277876    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rx46t\""
Oct 30 21:43:12.277920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.277883    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:43:12.278257 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.278242    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-8c5b8\""
Oct 30 21:43:12.279770 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.279755    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6s2r8\""
Oct 30 21:43:12.280905 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.280889    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:43:12.280947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.280903    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:43:12.280947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.280928    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:43:12.280947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.280936    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hdwjf\""
Oct 30 21:43:12.285191 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.285178    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:43:12.285242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.285203    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2ds7\""
Oct 30 21:43:12.285305 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.285292    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:43:12.286309 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.286286    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxkz7\""
Oct 30 21:43:12.286346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.286297    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.288849 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.288835    5510 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:43:12.312949 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.312926    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.313022 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.312956    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:43:12.313022 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.312991    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv9kv\""
Oct 30 21:43:12.313022 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313015    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:43:12.313128 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313038    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:43:12.313128 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313091    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-27snf\""
Oct 30 21:43:12.313196 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313135    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k8pd4\""
Oct 30 21:43:12.313357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313335    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:43:12.313476 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313457    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:43:12.313539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313488    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:43:12.313539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313510    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:43:12.313539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313521    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:43:12.313539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313528    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:43:12.313539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313523    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:43:12.313745 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313645    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ag5tqi2uauu0p\""
Oct 30 21:43:12.313745 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313647    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:43:12.313993 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.313979    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:43:12.322639 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.322622    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:43:12.322939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.322926    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:43:12.323063 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323053    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:43:12.323113 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323072    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:43:12.323113 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323080    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:43:12.323217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323126    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:43:12.323217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323136    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:43:12.323217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323150    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89d9x\""
Oct 30 21:43:12.323337 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323234    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:43:12.323337 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323300    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:43:12.323337 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323307    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:43:12.323467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323410    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k4gxb\""
Oct 30 21:43:12.323467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.323450    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:43:12.367989 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.367930    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:43:12.375769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375750    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.375849 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375776    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.375849 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375816    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.375849 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375833    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.376034 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375854    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4s5\" (UniqueName: \"kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.376034 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375917    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b36d9e-7722-4db1-acb4-7a75f0df705c-tmp-dir\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.376034 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.375994    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.376155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376059    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.376155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376077    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54348157-31ac-475e-90a1-32b963be27a3-original-pull-secret\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.376155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376096    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cni-binary-copy\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.376155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376098    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.376155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376070    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.376301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376151    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9185dde-8443-434d-9df7-27d7e287b372-tmp\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.376301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376174    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.376301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376218    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.376301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376235    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.376301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376257    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jml4r\" (UniqueName: \"kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:12.376662 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376641    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.376703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376684    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqr2\" (UniqueName: \"kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.376748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.376807 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376770    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.376851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376828    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.376851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376833    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.376920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376863    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.376920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.376894    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.377030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377014    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.377074 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377051    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.377120 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377100    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.377159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377113    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.377189 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377175    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.377231 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377194    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8b30f97-b15d-4c82-b314-2849193c3a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.377231 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377212    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.377293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377245    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.377293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.377353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377310    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.377353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377343    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.377420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377376    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.377420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377410    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.377481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377441    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-script-lib\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.377481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377455    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.377579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377478    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-tls\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.377579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377506    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.377579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377563    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnxl\" (UniqueName: \"kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:43:12.377670 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377598    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mffs\" (UniqueName: \"kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.377705 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377665    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465b6111-ca93-426c-9901-e452be8ba99e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.377742 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377707    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-config-volume\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.378004 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.377981    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.378105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378025    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.378105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378056    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.378105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378089    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378121    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378130    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465b6111-ca93-426c-9901-e452be8ba99e-data-volume\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378154    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378198    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378233    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.378261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378245    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-tuned\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378266    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378299    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378301    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378332    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378373    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378412    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378449    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.378519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378467    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b640ceb8-5397-4ff2-9428-ab03c9cf3474-konnectivity-ca\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378527    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378555    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378560    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378588    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378744    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-metrics-client-ca\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378802    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378841    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378864    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-grpc-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.378889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.378889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.379262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379081    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.379262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379205    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.379262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379242    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.379386 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379259    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.379386 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379285    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.379489 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379469    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-daemon-config\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.379596 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379574    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.379694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379681    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.379763 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379747    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.379876 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379859    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.379976 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9c2\" (UniqueName: \"kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:12.380064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380052    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:43:12.380153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380141    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:12.380240 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380228    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.380331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.380425 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380413    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.380511 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380497    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.380578 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380560    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.380651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380637    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.380744 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380732    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.380839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6dc066-7ed1-4e73-86ad-5455718c1106-cert\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:43:12.380963 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.379952    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3caeefbd-7ff9-4082-837d-48155be1d7b5-tmp\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.381012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.380863    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.381272 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381250    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.381372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381292    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.381372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381326    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:12.381495 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381382    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.381579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381554    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.381935 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381578    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.381935 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.381848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67a0fce0-c116-4eb1-b233-066ba56adb42-serviceca\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.382170 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382133    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.382170 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382133    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.382252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382221    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.382297 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382272    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.382346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382300    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshbk\" (UniqueName: \"kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.382346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382305    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099cdf15-a1f3-456c-bbf5-34b2363b6142-metrics-certs\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:12.382346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382321    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.382346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382333    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.382517 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382342    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-metrics-tls\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.382517 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382351    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.382848 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382822    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.382931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382868    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5kf\" (UniqueName: \"kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.382931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382897    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.383033 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382963    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:43:12.383033 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.382997    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.383153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383068    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-ovnkube-config\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.383153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383116    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.383153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383130    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.383153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383141    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.383153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383134    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465b6111-ca93-426c-9901-e452be8ba99e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383199    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383256    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383277    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b640ceb8-5397-4ff2-9428-ab03c9cf3474-agent-certs\") pod \"konnectivity-agent-9rtpn\" (UID: \"b640ceb8-5397-4ff2-9428-ab03c9cf3474\") " pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383317    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-tmp-dir\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.383357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383352    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383378    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383451    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383460    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9185dde-8443-434d-9df7-27d7e287b372-metrics-client-ca\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383481    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z99j\" (UniqueName: \"kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383514    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383537    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.383635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383612    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383715    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383760    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383770    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-tls\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383801    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383886    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.383904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383889    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383888    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-textfile\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383909    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383947    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383956    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7cq\" (UniqueName: \"kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.383983    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384001    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.384079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384038    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-env-overrides\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384117    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.384310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384212    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.384310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384237    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twznq\" (UniqueName: \"kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.384310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384238    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.384310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384288    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384323    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887zx\" (UniqueName: \"kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384384    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384414    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-accelerators-collector-config\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.384523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384497    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj992\" (UniqueName: \"kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384594    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.384708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384642    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.384708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384644    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4nt\" (UniqueName: \"kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.384818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384759    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.384896 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384883    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384932 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.384932 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384921    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd48\" (UniqueName: \"kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.384998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384936    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.384998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384933    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1012b47f-8e91-41ca-bc3e-8597142bce89-iptables-alerter-script\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.384998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.384975    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.385099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385007    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.385099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385035    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.385099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385061    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lqd\" (UniqueName: \"kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.385205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385114    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9185dde-8443-434d-9df7-27d7e287b372-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.385205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385139    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.385205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385164    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.385205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385185    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.385359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385239    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.385359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385259    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.385359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385281    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:12.385359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385297    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.385519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385479    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b30f97-b15d-4c82-b314-2849193c3a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.385609 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.385593    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab8d50c8-864a-4758-9ece-62c693d9cffd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:12.429174 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.429148    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnxl\" (UniqueName: \"kubernetes.io/projected/cf6dc066-7ed1-4e73-86ad-5455718c1106-kube-api-access-8tnxl\") pod \"ingress-canary-r9zbm\" (UID: \"cf6dc066-7ed1-4e73-86ad-5455718c1106\") " pod="openshift-ingress-canary/ingress-canary-r9zbm"
Oct 30 21:43:12.429260 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.429175    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4s5\" (UniqueName: \"kubernetes.io/projected/465b6111-ca93-426c-9901-e452be8ba99e-kube-api-access-fq4s5\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.431693 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.431677    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jml4r\" (UniqueName: \"kubernetes.io/projected/099cdf15-a1f3-456c-bbf5-34b2363b6142-kube-api-access-jml4r\") pod \"network-metrics-daemon-l84ww\" (UID: \"099cdf15-a1f3-456c-bbf5-34b2363b6142\") " pod="openshift-multus/network-metrics-daemon-l84ww"
Oct 30 21:43:12.432112 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.432092    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqr2\" (UniqueName: \"kubernetes.io/projected/d7b36d9e-7722-4db1-acb4-7a75f0df705c-kube-api-access-ltqr2\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.432252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.432233    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mffs\" (UniqueName: \"kubernetes.io/projected/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kube-api-access-4mffs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.432985 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.432969    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9c2\" (UniqueName: \"kubernetes.io/projected/ab8d50c8-864a-4758-9ece-62c693d9cffd-kube-api-access-bx9c2\") pod \"monitoring-plugin-77f4f76594-wp7wz\" (UID: \"ab8d50c8-864a-4758-9ece-62c693d9cffd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:12.433549 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.433530    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dslv\" (UniqueName: \"kubernetes.io/projected/7e23aa0b-cefb-4c96-b4a6-d48235deb03b-kube-api-access-4dslv\") pod \"network-check-target-68rh6\" (UID: \"7e23aa0b-cefb-4c96-b4a6-d48235deb03b\") " pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:12.436977 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.436960    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z99j\" (UniqueName: \"kubernetes.io/projected/c1a8572a-f8ad-4c98-8758-cc25c4823aaa-kube-api-access-8z99j\") pod \"dns-default-dmnk2\" (UID: \"c1a8572a-f8ad-4c98-8758-cc25c4823aaa\") " pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:12.436977 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.436967    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7cq\" (UniqueName: \"kubernetes.io/projected/67a0fce0-c116-4eb1-b233-066ba56adb42-kube-api-access-fq7cq\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.437664 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.437643    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"alertmanager-main-0\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:12.437763 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.437746    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5kf\" (UniqueName: \"kubernetes.io/projected/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-kube-api-access-sd5kf\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.438079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.438063    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshbk\" (UniqueName: \"kubernetes.io/projected/1012b47f-8e91-41ca-bc3e-8597142bce89-kube-api-access-lshbk\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.439383 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.439362    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887zx\" (UniqueName: \"kubernetes.io/projected/e8b30f97-b15d-4c82-b314-2849193c3a98-kube-api-access-887zx\") pod \"prometheus-operator-9cd89b74c-rf5bb\" (UID: \"e8b30f97-b15d-4c82-b314-2849193c3a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rf5bb"
Oct 30 21:43:12.439489 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.439413    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4nt\" (UniqueName: \"kubernetes.io/projected/3caeefbd-7ff9-4082-837d-48155be1d7b5-kube-api-access-fv4nt\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.439642 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.439620    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twznq\" (UniqueName: \"kubernetes.io/projected/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-kube-api-access-twznq\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.439730 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.439714    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj992\" (UniqueName: \"kubernetes.io/projected/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-kube-api-access-dj992\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.439837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.439823    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lqd\" (UniqueName: \"kubernetes.io/projected/a9185dde-8443-434d-9df7-27d7e287b372-kube-api-access-z2lqd\") pod \"thanos-querier-7d8d48d5b6-7rbd4\" (UID: \"a9185dde-8443-434d-9df7-27d7e287b372\") " pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:12.440084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.440070    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd48\" (UniqueName: \"kubernetes.io/projected/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-kube-api-access-wxd48\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.485757 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485731    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.485855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485761    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.485855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485794    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.485855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485816    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.485855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485840    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.485855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485850    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-sys-fs\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485861    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-k8s-cni-cncf-io\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485873    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485865    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485891    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-etc-kubernetes\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485905    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485928    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485935    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-root\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485948    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485976    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485993    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485998    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-multus-certs\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486002    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-bin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.485999    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486033    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-kubelet\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486034    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486042    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486075    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.486086 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486098    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysctl-conf\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486101    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486080    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465b6111-ca93-426c-9901-e452be8ba99e-crio-socket\") pod \"insights-runtime-extractor-79gfb\" (UID: \"465b6111-ca93-426c-9901-e452be8ba99e\") " pod="openshift-insights/insights-runtime-extractor-79gfb"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486130    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486141    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486157    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7b36d9e-7722-4db1-acb4-7a75f0df705c-hosts-file\") pod \"node-resolver-4zb2n\" (UID: \"d7b36d9e-7722-4db1-acb4-7a75f0df705c\") " pod="openshift-dns/node-resolver-4zb2n"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486149    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486195    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-system-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486197    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486229    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486172    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-system-cni-dir\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486254    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486279    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486277    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-slash\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486305    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1012b47f-8e91-41ca-bc3e-8597142bce89-host-slash\") pod \"iptables-alerter-ccs7w\" (UID: \"1012b47f-8e91-41ca-bc3e-8597142bce89\") " pod="openshift-network-operator/iptables-alerter-ccs7w"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486306    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486282    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-run-netns\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486336    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-ovn\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486366    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-host\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486371    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486388    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486344    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-sys\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486407    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486435    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486442    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-kubernetes\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486444    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-node-exporter-wtmp\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486457    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67a0fce0-c116-4eb1-b233-066ba56adb42-host\") pod \"node-ca-2cj8d\" (UID: \"67a0fce0-c116-4eb1-b233-066ba56adb42\") " pod="openshift-image-registry/node-ca-2cj8d"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486493    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486497    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486506    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-device-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486510    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-kubelet\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486534    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486550    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486565    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486551    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-dbus\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486593    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486609    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-cnibin\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486610    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-systemd-units\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486620    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486622    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-node-log\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486642    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-socket-dir-parent\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486660    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-run-netns\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486672    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486692    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-lib-modules\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486697    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.486710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486721    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486749    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486752    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-netd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486766    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf6b327-54b3-42dd-bf21-4c4ecd526af7-cnibin\") pod \"multus-additional-cni-plugins-ktk5z\" (UID: \"5bf6b327-54b3-42dd-bf21-4c4ecd526af7\") " pod="openshift-multus/multus-additional-cni-plugins-ktk5z"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486792    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486800    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486815    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-sysconfig\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486824    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486843    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-var-lib-kubelet\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486847    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486874    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-hostroot\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486873    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486908    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-host-var-lib-cni-multus\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486918    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486923    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-modprobe-d\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486945    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486967    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-cni-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486980    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.486993    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-run-systemd\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487029    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487060    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-run\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487085    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487093    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54348157-31ac-475e-90a1-32b963be27a3-kubelet-config\") pod \"global-pull-secret-syncer-wr6rh\" (UID: \"54348157-31ac-475e-90a1-32b963be27a3\") " pod="kube-system/global-pull-secret-syncer-wr6rh"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487095    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487117    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-log-socket\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487100    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487120    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69ef60f7-bf87-4e70-bf23-4671a0ea4f23-multus-conf-dir\") pod \"multus-7w2xx\" (UID: \"69ef60f7-bf87-4e70-bf23-4671a0ea4f23\") " pod="openshift-multus/multus-7w2xx"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487148    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487174    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-host-cni-bin\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487180    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487213    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487231    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-socket-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487236    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487261    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76a1da5e-7477-453d-bd89-cbe7f9e7b20a-sys\") pod \"node-exporter-65jnb\" (UID: \"76a1da5e-7477-453d-bd89-cbe7f9e7b20a\") " pod="openshift-monitoring/node-exporter-65jnb"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487263    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487282    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-var-lib-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487281    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3caeefbd-7ff9-4082-837d-48155be1d7b5-etc-systemd\") pod \"tuned-nhnkd\" (UID: \"3caeefbd-7ff9-4082-837d-48155be1d7b5\") " pod="openshift-cluster-node-tuning-operator/tuned-nhnkd"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487290    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487309    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487357    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8c9495d-4278-42d4-98dc-a2e8b1dc356d-registration-dir\") pod \"aws-ebs-csi-driver-node-7zgc7\" (UID: \"f8c9495d-4278-42d4-98dc-a2e8b1dc356d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zgc7"
Oct 30 21:43:12.488208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:12.487375    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c728bb-8670-4f08-a095-b0b9aa9adcc7-etc-openvswitch\") pod \"ovnkube-node-t6z7q\" (UID: \"f7c728bb-8670-4f08-a095-b0b9aa9adcc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:13.140903 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:13.140756    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:13.278944 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:13.278894    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:43:14.628836 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:14.628804    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dmnk2"
Oct 30 21:43:14.863060 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:14.863025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:15.081080 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:15.081049    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:15.714515 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:15.714484    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:16.404364 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:16.404340    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:16.404536 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:16.404525    5510 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:43:16.616220 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:16.616189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:16.859541 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:16.859509    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:17.126557 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:17.126489    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d8d48d5b6-7rbd4"
Oct 30 21:43:17.650760 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:17.650731    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wp7wz"
Oct 30 21:43:18.839062 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:18.839027    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:19.123795 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:19.123716    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:22.146569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:22.146493    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6z7q"
Oct 30 21:43:23.002842 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:23.002803    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:23.167308 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:23.167276    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:28.505799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:28.505745    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rtpn"
Oct 30 21:43:38.288248 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:38.288212    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:38.480481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:38.480436    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:41.281477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:41.281427    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:41.635723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:41.635648    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:43.277442 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:43.277403    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:43:43.320958 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.320929    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:43.321420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321382    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="alertmanager" containerID="cri-o://3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063" gracePeriod=120
Oct 30 21:43:43.321420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321402    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-metric" containerID="cri-o://0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4" gracePeriod=120
Oct 30 21:43:43.321630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321438    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="config-reloader" containerID="cri-o://2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509" gracePeriod=120
Oct 30 21:43:43.321630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321453    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-web" containerID="cri-o://e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007" gracePeriod=120
Oct 30 21:43:43.321630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321413    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="prom-label-proxy" containerID="cri-o://c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986" gracePeriod=120
Oct 30 21:43:43.321630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.321475    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy" containerID="cri-o://128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce" gracePeriod=120
Oct 30 21:43:43.571986 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.571964    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601172    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601251    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601292    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601330    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601410    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601445    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601487    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601603    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.602289 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.601681    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") pod \"700a06ab-4440-42e1-abcc-ca01af3ffd59\" (UID: \"700a06ab-4440-42e1-abcc-ca01af3ffd59\") "
Oct 30 21:43:43.605585 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.605553    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:43.606004 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.605859    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:43.606498 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.605828    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:43.606634 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.606153    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:43.608328 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.608262    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.608474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.608368    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:43.608542 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.608468    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "kube-api-access-s4tlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:43.608961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.608934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.609063 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.609048    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.609222 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.609185    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.609627 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.609608    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.612205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.612180    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.617424 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.617403    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config" pod "700a06ab-4440-42e1-abcc-ca01af3ffd59" (UID: "700a06ab-4440-42e1-abcc-ca01af3ffd59"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:43.703183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703150    5510 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-web-config\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703188    5510 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-main-db\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703206    5510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703223    5510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-main-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703240    5510 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703254    5510 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-tls-assets\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703267    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700a06ab-4440-42e1-abcc-ca01af3ffd59-metrics-client-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703281    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4tlr\" (UniqueName: \"kubernetes.io/projected/700a06ab-4440-42e1-abcc-ca01af3ffd59-kube-api-access-s4tlr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703296    5510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703310    5510 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-cluster-tls-config\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703324    5510 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-out\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703339    5510 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:43.703615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:43.703354    5510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/700a06ab-4440-42e1-abcc-ca01af3ffd59-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295086    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986" exitCode=0
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295110    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4" exitCode=0
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295119    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce" exitCode=0
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295124    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007" exitCode=0
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295130    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509" exitCode=0
Oct 30 21:43:44.295123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295136    5510 generic.go:356] "Generic (PLEG): container finished" podID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063" exitCode=0
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295162    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295193    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295229    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295260    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295285    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295296    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295310    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295336    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"}
Oct 30 21:43:44.295871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.295359    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"700a06ab-4440-42e1-abcc-ca01af3ffd59","Type":"ContainerDied","Data":"f7e1979c3914f58b04fba723889fef09ad4ac15ed972d85d438de513923c8cb8"}
Oct 30 21:43:44.302895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.302874    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.309868 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.309851    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.315979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.315962    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.322226 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.322080    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.328320 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.328304    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.342663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.342645    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.349154 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349139    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.349423 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.349403    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.349482 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349430    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.349482 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349458    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.349701 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.349684    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.349752 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349704    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.349752 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349717    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.349951 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.349936    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.349991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349956    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.349991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.349969    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.350172 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.350157    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.350213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350176    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.350213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350188    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.350392 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.350377    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.350430 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350396    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.350430 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350408    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.350592 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.350578    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.350630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350595    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.350630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350606    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.350772 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:43:44.350758    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.350833 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350788    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.350833 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350800    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.350974 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350959    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.351013 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.350974    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.351173 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351156    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.351212 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351174    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.351370 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351349    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.351411 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351372    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.351585 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351569    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.351585 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351584    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.351802 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351769    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.351862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.351802    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.352017 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352000    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.352070 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352016    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.352212 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352193    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.352265 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352212    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.352408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352391    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.352472 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352407    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.352603 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352586    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.352660 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352602    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.352814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352798    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.352814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.352814    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.353028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353011    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.353028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353028    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.353244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353226    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.353244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353242    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.353465 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353447    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.353518 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353466    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.353705 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353686    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.353757 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353705    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.353947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353930    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.353988 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.353947    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.354183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354166    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.354228 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354184    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.354382 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354367    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.354429 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354382    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.354557 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354544    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.354597 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354556    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.354737 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354717    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.354800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354737    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.354957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354939    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.354994 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.354957    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.355184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355155    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.355184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355175    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.355375 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355358    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.355470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355377    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.355618 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355601    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.355679 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355619    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.355839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355819    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.355886 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.355840    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.356044 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356029    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.356089 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356044    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.356252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356230    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.356252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356251    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.356451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356434    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.356451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356451    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.356662 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356647    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.356703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356662    5510 scope.go:122] "RemoveContainer" containerID="c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"
Oct 30 21:43:44.356893 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356870    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986"} err="failed to get container status \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": rpc error: code = NotFound desc = could not find container \"c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986\": container with ID starting with c671dbf054c49b34fd1696fd897b7adea63784730d18e70c9c637a4ba13e2986 not found: ID does not exist"
Oct 30 21:43:44.356939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.356894    5510 scope.go:122] "RemoveContainer" containerID="0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"
Oct 30 21:43:44.357121 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357105    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4"} err="failed to get container status \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": rpc error: code = NotFound desc = could not find container \"0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4\": container with ID starting with 0638160c1d885000e4d6a5df685f90c4539a31ae8930a950ad31195d937534f4 not found: ID does not exist"
Oct 30 21:43:44.357174 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357121    5510 scope.go:122] "RemoveContainer" containerID="128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"
Oct 30 21:43:44.357322 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357307    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce"} err="failed to get container status \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": rpc error: code = NotFound desc = could not find container \"128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce\": container with ID starting with 128f3b038cf1ea8d0bb7143e7f8b755423ef70d62b6daac7df0e8e453be329ce not found: ID does not exist"
Oct 30 21:43:44.357322 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357322    5510 scope.go:122] "RemoveContainer" containerID="e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"
Oct 30 21:43:44.357526 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357506    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007"} err="failed to get container status \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": rpc error: code = NotFound desc = could not find container \"e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007\": container with ID starting with e5b26d978e00ec7f640267f4e8a82fcfd9636ff40c4f0989582590cfa4c1f007 not found: ID does not exist"
Oct 30 21:43:44.357576 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357528    5510 scope.go:122] "RemoveContainer" containerID="2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"
Oct 30 21:43:44.357730 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357712    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509"} err="failed to get container status \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": rpc error: code = NotFound desc = could not find container \"2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509\": container with ID starting with 2480af6750bb2cd99205d33cd56e1744162fe6be70af92770b97396af35cb509 not found: ID does not exist"
Oct 30 21:43:44.357789 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357730    5510 scope.go:122] "RemoveContainer" containerID="3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"
Oct 30 21:43:44.357970 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357954    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063"} err="failed to get container status \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": rpc error: code = NotFound desc = could not find container \"3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063\": container with ID starting with 3a89fcefb5c808c1f22e4c41398e836f07df89c1a75f0daff9088a6f974cf063 not found: ID does not exist"
Oct 30 21:43:44.357970 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.357970    5510 scope.go:122] "RemoveContainer" containerID="db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"
Oct 30 21:43:44.358189 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.358173    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2"} err="failed to get container status \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": rpc error: code = NotFound desc = could not find container \"db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2\": container with ID starting with db0456dcbe4bbd0147b26e431b9244c48e6f5a11adaba26b268a66dae1a09ba2 not found: ID does not exist"
Oct 30 21:43:44.450128 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.450098    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:44.496205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496175    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:44.496375 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496357    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:44.496583 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496572    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-web"
Oct 30 21:43:44.496621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496584    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-web"
Oct 30 21:43:44.496621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496598    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="config-reloader"
Oct 30 21:43:44.496621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496604    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="config-reloader"
Oct 30 21:43:44.496621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496620    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="alertmanager"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496626    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="alertmanager"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496632    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="init-config-reloader"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496636    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="init-config-reloader"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496643    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="prom-label-proxy"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496647    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="prom-label-proxy"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496653    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-metric"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496657    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-metric"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496665    5510 cpu_manager.go:414] "containerMap: removing container" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496669    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496709    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-web"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496717    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="config-reloader"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496725    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="alertmanager"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496730    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="init-config-reloader"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496737    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="prom-label-proxy"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496741    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy-metric"
Oct 30 21:43:44.496748 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.496748    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" containerName="kube-rbac-proxy"
Oct 30 21:43:44.499749 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.499735    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.554407 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.554340    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:43:44.554407 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.554397    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89d9x\""
Oct 30 21:43:44.555199 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.555187    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:43:44.556121 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.556109    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:43:44.556183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.556170    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:43:44.556262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.556253    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:43:44.556441 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.556431    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:43:44.602050 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.602017    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:43:44.603276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.603262    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:43:44.603331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.603274    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:43:44.608501 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgn2\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-kube-api-access-stgn2\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608550 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608510    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-volume\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608550 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608535    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608634 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608552    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608680 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608628    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608773 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608750    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608835    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608892 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608883    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608925 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608904    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-out\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.608925 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608921    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-web-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.609000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.608937    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.609039 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.609030    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.609117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.609099    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.622696 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.622664    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:44.709756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709722    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.709756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709756    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-out\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709773    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-web-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709911    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.709994    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710045    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgn2\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-kube-api-access-stgn2\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710077    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-volume\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710128    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710134    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710161    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710192    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710594 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.710594 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.710127    5510 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:43:44.711033 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.711006    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.711198 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.711083    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1444bd0b-372f-4db4-9efe-5d330a550ad8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.713263 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713239    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.713831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713598    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.713831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713668    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-out\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.713831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713727    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-config-volume\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.713831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713758    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.714046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713849    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-web-config\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.714046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.714046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.713950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1444bd0b-372f-4db4-9efe-5d330a550ad8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.714432 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.714417    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.763584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.763559    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgn2\" (UniqueName: \"kubernetes.io/projected/1444bd0b-372f-4db4-9efe-5d330a550ad8-kube-api-access-stgn2\") pod \"alertmanager-main-0\" (UID: \"1444bd0b-372f-4db4-9efe-5d330a550ad8\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:44.816643 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:44.816592    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:43:45.121916 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:45.121833    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700a06ab-4440-42e1-abcc-ca01af3ffd59" path="/var/lib/kubelet/pods/700a06ab-4440-42e1-abcc-ca01af3ffd59/volumes"
Oct 30 21:43:45.235148 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:45.235116    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:43:45.238416 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:45.238389    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1444bd0b_372f_4db4_9efe_5d330a550ad8.slice/crio-c189ad18838aaa1fd0ead64ae92557a4de30f737d51a8e807f21b5c5d950200e WatchSource:0}: Error finding container c189ad18838aaa1fd0ead64ae92557a4de30f737d51a8e807f21b5c5d950200e: Status 404 returned error can't find the container with id c189ad18838aaa1fd0ead64ae92557a4de30f737d51a8e807f21b5c5d950200e
Oct 30 21:43:45.300306 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:45.300276    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"71d2df3dff2761b4ad1115749d84eda2ba319f764e935245274721a18f646174"}
Oct 30 21:43:45.300705 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:45.300358    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"c189ad18838aaa1fd0ead64ae92557a4de30f737d51a8e807f21b5c5d950200e"}
Oct 30 21:43:45.717565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:45.717533    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-68rh6"
Oct 30 21:43:46.305244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:46.305205    5510 generic.go:356] "Generic (PLEG): container finished" podID="1444bd0b-372f-4db4-9efe-5d330a550ad8" containerID="71d2df3dff2761b4ad1115749d84eda2ba319f764e935245274721a18f646174" exitCode=0
Oct 30 21:43:46.305629 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:46.305271    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerDied","Data":"71d2df3dff2761b4ad1115749d84eda2ba319f764e935245274721a18f646174"}
Oct 30 21:43:47.229195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.229153    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67569c9584-zz8p8"]
Oct 30 21:43:47.231228 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.231210    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.283656 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.283615    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:43:47.283807 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.283725    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:43:47.283807 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.283743    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:43:47.283981 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.283967    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:43:47.284044 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.284000    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ktjsr\""
Oct 30 21:43:47.312282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312258    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"d24803ba4a6012a306908431c5c8d8f3e8f0c94d7580df3c8ccb3ab47fa59c54"}
Oct 30 21:43:47.312591 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312313    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"0124729b7745ca4be97ef03d483fb76f5c12039896fec987cb5a15160b31c91e"}
Oct 30 21:43:47.312591 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312340    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"1fa3cbebc573ea5cdb7b5bb9bf69ec05d4a1171e7047e2a37d7aa7387ea5bfd6"}
Oct 30 21:43:47.312591 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312365    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"5ec4750f89f729f13dc45f264c5850c8fcfc7baed3ed2ea1f83160fa6e6334cc"}
Oct 30 21:43:47.312591 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312400    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"f6831cecf67a9a434088d356718bd3db83e7442302111bdd690fc289f779fab3"}
Oct 30 21:43:47.312591 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.312424    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1444bd0b-372f-4db4-9efe-5d330a550ad8","Type":"ContainerStarted","Data":"b59bc4114a2e8927bb0ec48ab14cf6a88b72ce1b9898058976744f79abdc4d85"}
Oct 30 21:43:47.329846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.329818    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.329960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.329856    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.329960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.329931    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-serving-certs-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.329960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.329958    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-federate-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.330065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.329996    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.330065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.330022    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-metrics-client-ca\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.330065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.330046    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.330065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.330063    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2g5\" (UniqueName: \"kubernetes.io/projected/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-kube-api-access-6c2g5\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.330211 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.330067    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:43:47.330211 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.330153    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:43:47.431043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431015    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-serving-certs-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431046    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-federate-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431086    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431170    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-metrics-client-ca\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431248    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431275    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2g5\" (UniqueName: \"kubernetes.io/projected/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-kube-api-access-6c2g5\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431521 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431502    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431618 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431587    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431838 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431815    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-serving-certs-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.431977 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.431949    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-metrics-client-ca\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.432140 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.432116    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.434057 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.434022    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-federate-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.434183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.434167    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.434299 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.434283    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-telemeter-client-tls\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.434395 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.434373    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-secret-telemeter-client\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.444005 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.443982    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67569c9584-zz8p8"]
Oct 30 21:43:47.483099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.483020    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4701430269999998 podStartE2EDuration="3.483008479s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.865452ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:47.48269076 +0000 UTC m=+36.466597635" watchObservedRunningTime="2026-10-30 21:43:47.483008479 +0000 UTC m=+36.466915352"
Oct 30 21:43:47.487486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.487456    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2g5\" (UniqueName: \"kubernetes.io/projected/1f3aadfa-fbef-4e8e-8ff2-4fb99373d517-kube-api-access-6c2g5\") pod \"telemeter-client-67569c9584-zz8p8\" (UID: \"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517\") " pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.543615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.543577    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8"
Oct 30 21:43:47.955354 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.955282    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67569c9584-zz8p8"]
Oct 30 21:43:47.958404 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:43:47.958376    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3aadfa_fbef_4e8e_8ff2_4fb99373d517.slice/crio-2236aa7844f4c35904d88c6a9432f4fab8788d0ab594c19c5cbe0a9dd809d1ab WatchSource:0}: Error finding container 2236aa7844f4c35904d88c6a9432f4fab8788d0ab594c19c5cbe0a9dd809d1ab: Status 404 returned error can't find the container with id 2236aa7844f4c35904d88c6a9432f4fab8788d0ab594c19c5cbe0a9dd809d1ab
Oct 30 21:43:47.960360 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:47.960342    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:43:48.317974 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:48.317935    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8" event={"ID":"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517","Type":"ContainerStarted","Data":"2236aa7844f4c35904d88c6a9432f4fab8788d0ab594c19c5cbe0a9dd809d1ab"}
Oct 30 21:43:48.805957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:48.805925    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:49.020339 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:49.020295    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:50.328523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:50.328492    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8" event={"ID":"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517","Type":"ContainerStarted","Data":"20a8057675e02882b22afbeb6bd225ced0b81b677e91c4ede3f02cc70fc91400"}
Oct 30 21:43:50.328860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:50.328585    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8" event={"ID":"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517","Type":"ContainerStarted","Data":"80e0d97bb8b4a233dab8e32cd4323bc88e8714a93298a3cc037ebc3bb856928d"}
Oct 30 21:43:50.328860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:50.328606    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8" event={"ID":"1f3aadfa-fbef-4e8e-8ff2-4fb99373d517","Type":"ContainerStarted","Data":"0b5ab986df0106488984c8191274f9f620432bd47681b4de510d2db5f55bbe7e"}
Oct 30 21:43:50.539676 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:50.539622    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67569c9584-zz8p8" podStartSLOduration=1.3707488749999999 podStartE2EDuration="3.539610063s" totalImagesPullingTime="2.168861188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:50.493410311 +0000 UTC m=+39.477317181" watchObservedRunningTime="2026-10-30 21:43:50.539610063 +0000 UTC m=+39.523516936"
Oct 30 21:43:51.251686 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:51.251620    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:51.483028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:51.482992    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:43:51.645392 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:43:51.645310    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:08.953690 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:08.953657    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:09.106902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:09.106871    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:13.278902 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:44:13.278867    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:44:13.629841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:13.629745    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:13.804083 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:13.804051    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:14.060292 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:14.060262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:14.225588 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:14.225543    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:16.917494 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:16.917457    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:21.623308 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:21.623271    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:21.788531 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:21.788493    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:33.721380 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:33.721347    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:33.875769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:44:33.875687    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:44:43.278647 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:44:43.278612    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:45:13.278535 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:45:13.278449    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:45:17.258214 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:17.258173    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:17.429135 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:17.429100    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:32.613441 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:32.613403    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:43.279125 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:45:43.279088    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:45:51.229254 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:51.229216    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:51.427226 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:51.427191    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:52.339545 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.339505    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"]
Oct 30 21:45:52.342602 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.342579    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.436842 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.436802    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.437011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.436900    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.437011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.436926    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hvkqs\""
Oct 30 21:45:52.437547 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.437531    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"]
Oct 30 21:45:52.441297 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.441278    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz85f\" (UniqueName: \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.441346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.441318    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.441380 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.441358    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.522857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.522713    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:45:52.525909 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.525887    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.542540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.542505    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.542687 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.542594    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.542687 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.542660    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz85f\" (UniqueName: \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.542937 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.542914    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.543012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.542992    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.577054 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.577017    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.577206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.577123    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:45:52.577206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.577136    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.581826 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.581799    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:52.584394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.584373    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:45:52.595691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.595638    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:45:52.595691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.595637    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:45:52.595691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.595656    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:45:52.595691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.595674    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dwlzg\""
Oct 30 21:45:52.598141 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.598120    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz85f\" (UniqueName: \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.643068 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643031    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643112    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643178    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643388 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643315    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643388 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643344    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvhc\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643499 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643426    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.643499 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.643483    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.657108 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.657076    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:45:52.740955 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.740921    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:52.743912 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.743892    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.743997 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.743936    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvhc\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.743997 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.743957    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.744102 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.744079    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.744153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.744118    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.744213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.744168    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.744213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.744197    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.744797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.744756    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.746233 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.746206    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.746326 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.746251    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.746397 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.746375    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.746486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.746466    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.797362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.797327    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.797494 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.797379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvhc\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\") pod \"istiod-openshift-gateway-7876bfcb74-4f65d\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:52.837230 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:52.837209    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:53.016916 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:53.016880    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"]
Oct 30 21:45:53.020849 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:45:53.020822    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bef00f_f482_4d9f_8f71_1fb81503c5a2.slice/crio-ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d WatchSource:0}: Error finding container ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d: Status 404 returned error can't find the container with id ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d
Oct 30 21:45:53.196431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:53.196348    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:45:53.199264 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:45:53.199227    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfabbb4_81b3_442c_b9a4_69e4206308ce.slice/crio-e8c0294d4b7ed3316bd111fdfa05cac5821803267269597437ae95f5b4420020 WatchSource:0}: Error finding container e8c0294d4b7ed3316bd111fdfa05cac5821803267269597437ae95f5b4420020: Status 404 returned error can't find the container with id e8c0294d4b7ed3316bd111fdfa05cac5821803267269597437ae95f5b4420020
Oct 30 21:45:53.574466 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:53.574437    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:53.740226 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:53.740189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerStarted","Data":"ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d"}
Oct 30 21:45:53.741543 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:53.741513    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" event={"ID":"abfabbb4-81b3-442c-b9a4-69e4206308ce","Type":"ContainerStarted","Data":"e8c0294d4b7ed3316bd111fdfa05cac5821803267269597437ae95f5b4420020"}
Oct 30 21:45:57.574961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:57.574926    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:45:57.575344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:57.575002    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:45:57.762996 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:57.762951    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" event={"ID":"abfabbb4-81b3-442c-b9a4-69e4206308ce","Type":"ContainerStarted","Data":"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"}
Oct 30 21:45:57.763184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:57.763036    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:45:57.973856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:57.973773    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" podStartSLOduration=1.600994461 podStartE2EDuration="5.973760977s" totalImagesPullingTime="4.372766516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:57.973561613 +0000 UTC m=+166.957468489" watchObservedRunningTime="2026-10-30 21:45:57.973760977 +0000 UTC m=+166.957667852"
Oct 30 21:45:58.768291 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:58.768247    5510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4f65d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:45:58.768690 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:58.768310    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:45:59.065549 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:59.065477    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:45:59.231915 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:45:59.231881    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.068831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:00.068742    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.366401 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:00.366368    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.511720 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:00.511678    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:00.777577 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:00.777544    5510 generic.go:356] "Generic (PLEG): container finished" podID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerID="95572d958389cb2b43330f948245d4dbe9c6abf64f6e1c0c096c5a8d06a3326d" exitCode=0
Oct 30 21:46:00.777810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:00.777596    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerDied","Data":"95572d958389cb2b43330f948245d4dbe9c6abf64f6e1c0c096c5a8d06a3326d"}
Oct 30 21:46:01.397147 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:01.397105    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:01.767619 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:01.767591    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:46:04.233825 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:04.233734    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:04.793431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:04.793396    5510 generic.go:356] "Generic (PLEG): container finished" podID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerID="ad061133fbce0ab1f4715ea594df48087576de2d20cad0d11f4ac914bbc0c02d" exitCode=0
Oct 30 21:46:04.793633 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:04.793460    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerDied","Data":"ad061133fbce0ab1f4715ea594df48087576de2d20cad0d11f4ac914bbc0c02d"}
Oct 30 21:46:11.821560 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:11.821524    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerStarted","Data":"9a91237ff9463e3c0985c4db139b1aec236c45c40157f69e2793f32014bcc600"}
Oct 30 21:46:11.931098 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:11.931052    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" podStartSLOduration=3.607529499 podStartE2EDuration="19.931035701s" totalImagesPullingTime="16.323506202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.930031425 +0000 UTC m=+180.913938301" watchObservedRunningTime="2026-10-30 21:46:11.931035701 +0000 UTC m=+180.914942571"
Oct 30 21:46:12.827389 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:12.827356    5510 generic.go:356] "Generic (PLEG): container finished" podID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerID="9a91237ff9463e3c0985c4db139b1aec236c45c40157f69e2793f32014bcc600" exitCode=0
Oct 30 21:46:12.827797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:12.827440    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerDied","Data":"9a91237ff9463e3c0985c4db139b1aec236c45c40157f69e2793f32014bcc600"}
Oct 30 21:46:13.278996 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:46:13.278954    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:46:14.011259 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.011228    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:46:14.113256 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.113178    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\") pod \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") "
Oct 30 21:46:14.113256 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.113225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\" (UniqueName: \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\") pod \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") "
Oct 30 21:46:14.113256 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.113258    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\") pod \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\" (UID: \"c9bef00f-f482-4d9f-8f71-1fb81503c5a2\") "
Oct 30 21:46:14.113883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.113843    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle" pod "c9bef00f-f482-4d9f-8f71-1fb81503c5a2" (UID: "c9bef00f-f482-4d9f-8f71-1fb81503c5a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:14.115472 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.115444    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f" pod "c9bef00f-f482-4d9f-8f71-1fb81503c5a2" (UID: "c9bef00f-f482-4d9f-8f71-1fb81503c5a2"). InnerVolumeSpecName "kube-api-access-mz85f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:14.117900 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.117856    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util" pod "c9bef00f-f482-4d9f-8f71-1fb81503c5a2" (UID: "c9bef00f-f482-4d9f-8f71-1fb81503c5a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:14.214159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.214117    5510 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-bundle\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:14.214159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.214154    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz85f\" (UniqueName: \"kubernetes.io/projected/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-kube-api-access-mz85f\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:14.214159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.214170    5510 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9bef00f-f482-4d9f-8f71-1fb81503c5a2-util\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:14.836709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.836676    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2" event={"ID":"c9bef00f-f482-4d9f-8f71-1fb81503c5a2","Type":"ContainerDied","Data":"ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d"}
Oct 30 21:46:14.836900 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.836727    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1ed9d0caa5b70b850309f6a924379b8359daf78791ca548132f35735df237d"
Oct 30 21:46:14.836900 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:14.836684    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckc8j2"
Oct 30 21:46:17.589927 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:17.589895    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:19.488663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:19.488630    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:41.797570 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:41.797535    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:41.946099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:41.946067    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:42.652345 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.652309    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:42.947987 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.947916    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"]
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948229    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="pull"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948243    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="pull"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948267    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="util"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948273    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="util"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948286    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="extract"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948291    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="extract"
Oct 30 21:46:42.948340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.948342    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bef00f-f482-4d9f-8f71-1fb81503c5a2" containerName="extract"
Oct 30 21:46:42.951313 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:42.951297    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026271 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026237    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzkf\" (UniqueName: \"kubernetes.io/projected/af7efdf1-9085-452d-a8d4-7478550166d0-kube-api-access-9fzkf\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026279    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026304    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026374    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026426    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.026663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.026554    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.048114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048091    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:46:43.048223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048129    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:46:43.048223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048128    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:46:43.048223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048162    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:46:43.048223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048176    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5cc62\""
Oct 30 21:46:43.048366 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.048272    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:46:43.049145 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.049128    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"]
Oct 30 21:46:43.050274 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.050255    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.127239 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127216    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127252    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzkf\" (UniqueName: \"kubernetes.io/projected/af7efdf1-9085-452d-a8d4-7478550166d0-kube-api-access-9fzkf\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127276    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127313    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127336    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127376    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.127703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.127675    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-manifests\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.129520 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.129497    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af7efdf1-9085-452d-a8d4-7478550166d0-tmp\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.129907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.129884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.130271 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.130251    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-metrics-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.130317 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.130258    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af7efdf1-9085-452d-a8d4-7478550166d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.183699 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.183672    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzkf\" (UniqueName: \"kubernetes.io/projected/af7efdf1-9085-452d-a8d4-7478550166d0-kube-api-access-9fzkf\") pod \"opendatahub-operator-controller-manager-8677f9fb98-9gbwk\" (UID: \"af7efdf1-9085-452d-a8d4-7478550166d0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.222973 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.222943    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.265462 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.265439    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:43.277983 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:46:43.277958    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:46:43.417244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.417208    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.588238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.588173    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.756474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.756443    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"]
Oct 30 21:46:43.764890 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:46:43.764862    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7efdf1_9085_452d_a8d4_7478550166d0.slice/crio-577293ab58ea67188da3275fade309952a756d22e368b64c4fbb8f7633c566a3 WatchSource:0}: Error finding container 577293ab58ea67188da3275fade309952a756d22e368b64c4fbb8f7633c566a3: Status 404 returned error can't find the container with id 577293ab58ea67188da3275fade309952a756d22e368b64c4fbb8f7633c566a3
Oct 30 21:46:43.937195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:43.937118    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk" event={"ID":"af7efdf1-9085-452d-a8d4-7478550166d0","Type":"ContainerStarted","Data":"577293ab58ea67188da3275fade309952a756d22e368b64c4fbb8f7633c566a3"}
Oct 30 21:46:44.984166 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:44.984134    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.139933 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:45.139895    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:46.015084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:46.015038    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:46.170903 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:46.170866    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:47.955170 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:47.955084    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk" event={"ID":"af7efdf1-9085-452d-a8d4-7478550166d0","Type":"ContainerStarted","Data":"b83d7d36dc9c107c24e5a6aa0927850c46bf6e89ce750b64d63a06d83a6e059a"}
Oct 30 21:46:48.960340 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:48.960302    5510 generic.go:356] "Generic (PLEG): container finished" podID="af7efdf1-9085-452d-a8d4-7478550166d0" containerID="b83d7d36dc9c107c24e5a6aa0927850c46bf6e89ce750b64d63a06d83a6e059a" exitCode=0
Oct 30 21:46:48.960707 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:48.960368    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk" event={"ID":"af7efdf1-9085-452d-a8d4-7478550166d0","Type":"ContainerDied","Data":"b83d7d36dc9c107c24e5a6aa0927850c46bf6e89ce750b64d63a06d83a6e059a"}
Oct 30 21:46:49.965571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:49.965544    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk" event={"ID":"af7efdf1-9085-452d-a8d4-7478550166d0","Type":"ContainerStarted","Data":"902a83e0d819a1fed7ead638dfd5efc3bfd71c9f1c8d8c159297d106d31b83a1"}
Oct 30 21:46:49.965978 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:49.965691    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:46:50.244709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:50.244659    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk" podStartSLOduration=3.078727108 podStartE2EDuration="8.244645631s" totalImagesPullingTime="4.738156689s" totalInitContainerRuntime="427.761834ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:50.198706704 +0000 UTC m=+219.182613579" watchObservedRunningTime="2026-10-30 21:46:50.244645631 +0000 UTC m=+219.228552506"
Oct 30 21:46:51.579538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:51.579503    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.093864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:53.093822    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.767939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:53.767909    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.975920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:54.975888    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:55.876651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:55.876622    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.092425 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:56.092380    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.301112 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:56.301080    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.978888 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:46:56.978855    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:00.972100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:00.972070    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8677f9fb98-9gbwk"
Oct 30 21:47:03.088414 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:03.088379    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:03.265570 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:03.265530    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:03.452601 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:03.452527    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:04.612551 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:04.612518    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:04.888125 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:04.888046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:05.993632 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:05.993597    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.203960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:06.203925    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.416074 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:06.415982    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.645177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:06.645147    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.225987 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:08.225953    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.821348 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:08.821316    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:10.614899 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:10.614865    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:11.463206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:11.463169    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:12.609014 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:12.608980    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:12.910971 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:12.910893    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:13.279314 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:47:13.279281    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:47:13.359097 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:13.359068    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:14.026478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:14.026445    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:14.255013 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:14.254972    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:14.647020 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:14.646981    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:14.873482 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:14.873439    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:15.209837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:15.209800    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:15.428627 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:15.428593    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:16.273450 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.273415    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:16.469296 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.469262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:16.608137 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.608066    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78776654bd-gpv65"]
Oct 30 21:47:16.614314 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.614266    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.655840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.655809    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:16.665723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.665691    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:47:16.693316 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.693275    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51094e-a499-404c-bbba-0e4372d86ffd-tls-certs\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.693438 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.693335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d51094e-a499-404c-bbba-0e4372d86ffd-tmp\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.693438 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.693393    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9sm\" (UniqueName: \"kubernetes.io/projected/0d51094e-a499-404c-bbba-0e4372d86ffd-kube-api-access-pg9sm\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.711116 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.711088    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:47:16.711332 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.711318    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z98gw\""
Oct 30 21:47:16.711971 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.711953    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78776654bd-gpv65"]
Oct 30 21:47:16.794007 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.793974    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51094e-a499-404c-bbba-0e4372d86ffd-tls-certs\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.794151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.794020    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d51094e-a499-404c-bbba-0e4372d86ffd-tmp\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.794151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.794039    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9sm\" (UniqueName: \"kubernetes.io/projected/0d51094e-a499-404c-bbba-0e4372d86ffd-kube-api-access-pg9sm\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.796257 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.796231    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d51094e-a499-404c-bbba-0e4372d86ffd-tmp\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.796476 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.796455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d51094e-a499-404c-bbba-0e4372d86ffd-tls-certs\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.846840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.846803    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:16.847619 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.847594    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9sm\" (UniqueName: \"kubernetes.io/projected/0d51094e-a499-404c-bbba-0e4372d86ffd-kube-api-access-pg9sm\") pod \"kube-auth-proxy-78776654bd-gpv65\" (UID: \"0d51094e-a499-404c-bbba-0e4372d86ffd\") " pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:16.926268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:16.926200    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65"
Oct 30 21:47:17.017496 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.017465    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:17.209830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.209800    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:17.309727 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.309690    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78776654bd-gpv65"]
Oct 30 21:47:17.317815 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:47:17.317763    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d51094e_a499_404c_bbba_0e4372d86ffd.slice/crio-7126529762adc905a48273446fa3f0a1d2ff39c894479d7df883768809076c2d WatchSource:0}: Error finding container 7126529762adc905a48273446fa3f0a1d2ff39c894479d7df883768809076c2d: Status 404 returned error can't find the container with id 7126529762adc905a48273446fa3f0a1d2ff39c894479d7df883768809076c2d
Oct 30 21:47:17.424661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.424630    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:17.604175 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.604096    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:17.983171 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:17.983143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:18.073403 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:18.073371    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65" event={"ID":"0d51094e-a499-404c-bbba-0e4372d86ffd","Type":"ContainerStarted","Data":"7126529762adc905a48273446fa3f0a1d2ff39c894479d7df883768809076c2d"}
Oct 30 21:47:18.388668 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:18.388585    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:18.560996 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:18.560962    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:18.812273 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:18.812211    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:19.008467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:19.008422    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:19.179608 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:19.179520    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:19.608002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:19.607969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:19.834095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:19.834038    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:20.356505 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.356471    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 21:47:20.360335 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.360308    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.396138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.396109    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:20.411275 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.411254    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:47:20.424422 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.424392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkh2\" (UniqueName: \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.424561 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.424430    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.439653 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.439633    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:47:20.439712 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.439635    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fxvwt\""
Oct 30 21:47:20.456499 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.456474    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 21:47:20.525391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.525364    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkh2\" (UniqueName: \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.525525 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.525395    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.525708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.525692    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.586909 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.586884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkh2\" (UniqueName: \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\") pod \"kuadrant-operator-controller-manager-678d87766f-fx5hj\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:20.620501 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.620442    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:20.674028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:20.674003    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:21.086080 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:21.086043    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65" event={"ID":"0d51094e-a499-404c-bbba-0e4372d86ffd","Type":"ContainerStarted","Data":"92dfb407d2aa05a23f356bbad84a1989eaaaa61cd2ce3dc621b1978d51b850be"}
Oct 30 21:47:21.146710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:21.146678    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 21:47:21.191013 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:47:21.190986    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f484336_ce43_403b_9024_25f71d813905.slice/crio-f0ee5db95f3947c4740eee05ec4857fc7dc80f659d244affc9d90cbcc5c87e43 WatchSource:0}: Error finding container f0ee5db95f3947c4740eee05ec4857fc7dc80f659d244affc9d90cbcc5c87e43: Status 404 returned error can't find the container with id f0ee5db95f3947c4740eee05ec4857fc7dc80f659d244affc9d90cbcc5c87e43
Oct 30 21:47:21.193908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:21.193858    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78776654bd-gpv65" podStartSLOduration=2.085751901 podStartE2EDuration="5.193845053s" totalImagesPullingTime="3.108093152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:21.192959196 +0000 UTC m=+250.176866072" watchObservedRunningTime="2026-10-30 21:47:21.193845053 +0000 UTC m=+250.177751929"
Oct 30 21:47:22.090789 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:22.090741    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" event={"ID":"7f484336-ce43-403b-9024-25f71d813905","Type":"ContainerStarted","Data":"f0ee5db95f3947c4740eee05ec4857fc7dc80f659d244affc9d90cbcc5c87e43"}
Oct 30 21:47:24.155541 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:24.155472    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:26.109800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:26.109753    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" event={"ID":"7f484336-ce43-403b-9024-25f71d813905","Type":"ContainerStarted","Data":"460c2a27231138acdee0e6dacdeec96da2dc7a61b02b5e53b3ee2d18cef2135a"}
Oct 30 21:47:26.110157 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:26.109897    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:26.320979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:26.320879    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" podStartSLOduration=1.509868464 podStartE2EDuration="6.320862629s" totalImagesPullingTime="4.810994165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:26.319917339 +0000 UTC m=+255.303824215" watchObservedRunningTime="2026-10-30 21:47:26.320862629 +0000 UTC m=+255.304769499"
Oct 30 21:47:26.589622 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:26.589546    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:26.762409 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:26.762352    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:29.898105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:29.898056    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:30.071057 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:30.071026    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:30.242611 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:30.242581    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:32.092012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:32.091980    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:32.383732 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:32.383655    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.147054 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:35.147019    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:35.452525 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:35.452433    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:37.116059 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:37.116027    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 21:47:40.110837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:40.110804    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:40.302016 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:40.301984    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:41.264188 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:41.264151    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:41.499678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:41.499647    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:43.277648 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:47:43.277612    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:47:46.608684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:46.608636    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:46.901797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:46.901711    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:53.102300 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:53.102216    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:53.324663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:53.324628    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:55.955843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:55.955810    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:56.142452 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:56.142420    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:56.304960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:56.304925    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:59.154723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.154692    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:47:59.158662 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.158643    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.207305 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.207279    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:47:59.253138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.253110    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v7c7z\""
Oct 30 21:47:59.253902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.253868    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:47:59.346538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.346505    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe39f822-603c-4dfd-b2bc-2a218e42dffa-config-file\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.346724 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.346572    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh69\" (UniqueName: \"kubernetes.io/projected/fe39f822-603c-4dfd-b2bc-2a218e42dffa-kube-api-access-dxh69\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.435379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.435311    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:47:59.447839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.447812    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh69\" (UniqueName: \"kubernetes.io/projected/fe39f822-603c-4dfd-b2bc-2a218e42dffa-kube-api-access-dxh69\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.447953 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.447866    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe39f822-603c-4dfd-b2bc-2a218e42dffa-config-file\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.448481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.448464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe39f822-603c-4dfd-b2bc-2a218e42dffa-config-file\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.500508 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.500480    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh69\" (UniqueName: \"kubernetes.io/projected/fe39f822-603c-4dfd-b2bc-2a218e42dffa-kube-api-access-dxh69\") pod \"limitador-limitador-7c9b87d574-rqcjz\" (UID: \"fe39f822-603c-4dfd-b2bc-2a218e42dffa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:47:59.770416 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:47:59.770386    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:48:00.134476 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:00.134394    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:48:00.142489 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:48:00.142461    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe39f822_603c_4dfd_b2bc_2a218e42dffa.slice/crio-b18e51fca3035fd4e29a35536a4573d17a983fd59ac42728813b0e44325bb11f WatchSource:0}: Error finding container b18e51fca3035fd4e29a35536a4573d17a983fd59ac42728813b0e44325bb11f: Status 404 returned error can't find the container with id b18e51fca3035fd4e29a35536a4573d17a983fd59ac42728813b0e44325bb11f
Oct 30 21:48:00.242598 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:00.242563    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz" event={"ID":"fe39f822-603c-4dfd-b2bc-2a218e42dffa","Type":"ContainerStarted","Data":"b18e51fca3035fd4e29a35536a4573d17a983fd59ac42728813b0e44325bb11f"}
Oct 30 21:48:01.307319 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:01.307279    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:01.495751 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:01.495706    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:01.727238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:01.727204    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:01.891839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:01.891802    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:02.053459 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:02.053371    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:03.257415 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:03.257381    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz" event={"ID":"fe39f822-603c-4dfd-b2bc-2a218e42dffa","Type":"ContainerStarted","Data":"fed8cc1eaec672b936fb6fd1c7c73d8772d86731b8de9ac2e736480246710df1"}
Oct 30 21:48:03.257878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:03.257635    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:48:03.367180 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:03.367128    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz" podStartSLOduration=2.126994567 podStartE2EDuration="4.367113706s" totalImagesPullingTime="2.240119139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:03.366123649 +0000 UTC m=+292.350030525" watchObservedRunningTime="2026-10-30 21:48:03.367113706 +0000 UTC m=+292.351020581"
Oct 30 21:48:08.369165 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:08.369124    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:08.544303 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:08.544268    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:11.149350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:11.149318    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:48:11.151585 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:11.151563    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:48:13.278738 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:48:13.278700    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:48:14.263031 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:14.263002    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"
Oct 30 21:48:14.969566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:14.969526    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:15.153421 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:15.153387    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:17.927408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:17.927365    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:18.140165 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:18.140132    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:18.319809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:18.319742    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:19.081459 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:19.081408    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:40.681722 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:40.681680    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:40.847623 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:40.847590    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:41.028973 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:41.028942    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:41.178752 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:41.178717    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:42.551621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.551587    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"]
Oct 30 21:48:42.555513 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.555493    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.652495 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.652469    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"]
Oct 30 21:48:42.662685 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662654    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.662862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662692    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.662862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662756    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.662862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662775    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2pm\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-kube-api-access-9q2pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.663046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662868    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.663046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662905    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.663046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.662988    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/709595ee-db7b-4df3-be34-fa8a572e5973-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763636    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763668    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763828    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2pm\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-kube-api-access-9q2pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763873    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.763922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763905    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.764073 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.763964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/709595ee-db7b-4df3-be34-fa8a572e5973-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.764556 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.764532    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.766098 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.766067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.766210 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.766124    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/709595ee-db7b-4df3-be34-fa8a572e5973-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.766210 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.766182    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.766299 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.766233    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/709595ee-db7b-4df3-be34-fa8a572e5973-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.817347 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.817287    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.817442 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.817374    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2pm\" (UniqueName: \"kubernetes.io/projected/709595ee-db7b-4df3-be34-fa8a572e5973-kube-api-access-9q2pm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w8v2q\" (UID: \"709595ee-db7b-4df3-be34-fa8a572e5973\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:42.867095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:42.867071    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:43.278739 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:48:43.278703    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:48:43.289034 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.289010    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"]
Oct 30 21:48:43.297452 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:48:43.297429    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709595ee_db7b_4df3_be34_fa8a572e5973.slice/crio-63a1c68db3f1941516712354dabe57c79263876d878cc4820fcb002856b7b429 WatchSource:0}: Error finding container 63a1c68db3f1941516712354dabe57c79263876d878cc4820fcb002856b7b429: Status 404 returned error can't find the container with id 63a1c68db3f1941516712354dabe57c79263876d878cc4820fcb002856b7b429
Oct 30 21:48:43.299463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.299435    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:48:43.299589 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.299506    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:48:43.413516 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.413483    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q" event={"ID":"709595ee-db7b-4df3-be34-fa8a572e5973","Type":"ContainerStarted","Data":"99e4a9ef0b54643effdd9aacb32ea9b03e047082486571ddb83bb2c2d6ad8f77"}
Oct 30 21:48:43.413686 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.413532    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q" event={"ID":"709595ee-db7b-4df3-be34-fa8a572e5973","Type":"ContainerStarted","Data":"63a1c68db3f1941516712354dabe57c79263876d878cc4820fcb002856b7b429"}
Oct 30 21:48:43.413686 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.413615    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:43.531479 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:43.531370    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q" podStartSLOduration=1.5313504980000001 podStartE2EDuration="1.531350498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:43.531013048 +0000 UTC m=+332.514919924" watchObservedRunningTime="2026-10-30 21:48:43.531350498 +0000 UTC m=+332.515257377"
Oct 30 21:48:44.419562 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:44.419533    5510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-w8v2q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:48:44.419960 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:44.419583    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q" podUID="709595ee-db7b-4df3-be34-fa8a572e5973" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:48:45.327873 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:45.327843    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:45.486290 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:45.486240    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:46.706754 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:46.706718    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:46.950025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:46.949993    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:47.248136 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.248101    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:47.419164 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.419136    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w8v2q"
Oct 30 21:48:47.506404 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.506331    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:47.578349 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.578324    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:48:47.578546 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.578524    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerName="discovery" containerID="cri-o://2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812" gracePeriod=30
Oct 30 21:48:47.684830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.684761    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:47.821372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.821349    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:48:47.907932 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.907902    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908071 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.907937    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908071 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.907972    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908071 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.908043    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908224 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.908161    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908275 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.908255    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908333 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.908309    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\") pod \"abfabbb4-81b3-442c-b9a4-69e4206308ce\" (UID: \"abfabbb4-81b3-442c-b9a4-69e4206308ce\") "
Oct 30 21:48:47.908537 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.908511    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:47.910363 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910342    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "kube-api-access-8zvhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:47.910467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910441    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:47.910535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910505    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:47.910599 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910550    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:47.910599 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910589    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:47.910851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:47.910823    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token" pod "abfabbb4-81b3-442c-b9a4-69e4206308ce" (UID: "abfabbb4-81b3-442c-b9a4-69e4206308ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:48.009151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009123    5510 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-kubeconfig\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009149    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009159    5510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-dns-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009168    5510 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfabbb4-81b3-442c-b9a4-69e4206308ce-local-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009177    5510 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/abfabbb4-81b3-442c-b9a4-69e4206308ce-cacerts\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009187    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvhc\" (UniqueName: \"kubernetes.io/projected/abfabbb4-81b3-442c-b9a4-69e4206308ce-kube-api-access-8zvhc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.009362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.009196    5510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/abfabbb4-81b3-442c-b9a4-69e4206308ce-istio-csr-ca-configmap\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:48.438678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.438639    5510 generic.go:356] "Generic (PLEG): container finished" podID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerID="2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812" exitCode=0
Oct 30 21:48:48.438865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.438685    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"
Oct 30 21:48:48.438865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.438688    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" event={"ID":"abfabbb4-81b3-442c-b9a4-69e4206308ce","Type":"ContainerDied","Data":"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"}
Oct 30 21:48:48.438865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.438832    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d" event={"ID":"abfabbb4-81b3-442c-b9a4-69e4206308ce","Type":"ContainerDied","Data":"e8c0294d4b7ed3316bd111fdfa05cac5821803267269597437ae95f5b4420020"}
Oct 30 21:48:48.438865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.438858    5510 scope.go:122] "RemoveContainer" containerID="2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"
Oct 30 21:48:48.447218 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.447197    5510 scope.go:122] "RemoveContainer" containerID="2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"
Oct 30 21:48:48.447510 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:48:48.447487    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812\": container with ID starting with 2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812 not found: ID does not exist" containerID="2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"
Oct 30 21:48:48.447572 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.447516    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812"} err="failed to get container status \"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812\": rpc error: code = NotFound desc = could not find container \"2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812\": container with ID starting with 2762146aaf0e05dd6a8e034fc7294e2be9a72f63229218f08c86184e3cae1812 not found: ID does not exist"
Oct 30 21:48:48.484195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.484168    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"]
Oct 30 21:48:48.484528 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.484516    5510 cpu_manager.go:414] "containerMap: removing container" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerName="discovery"
Oct 30 21:48:48.484571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.484529    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerName="discovery"
Oct 30 21:48:48.484605 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.484588    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" containerName="discovery"
Oct 30 21:48:48.489244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.489228    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.539527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.539499    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:48:48.539527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.539518    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-5ggpj\""
Oct 30 21:48:48.540796 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.540750    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"]
Oct 30 21:48:48.550573 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.550552    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:48:48.550675 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.550660    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:48:48.550732 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.550722    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:48:48.550910 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.550890    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:48:48.615703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.615676    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b073d99-dda0-4c19-aa4c-99b181577916-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.615862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.615753    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.615862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.615814    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzxr\" (UniqueName: \"kubernetes.io/projected/4b073d99-dda0-4c19-aa4c-99b181577916-kube-api-access-9nzxr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.615862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.615850    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.653242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.653206    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:48.716276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.716248    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b073d99-dda0-4c19-aa4c-99b181577916-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.716412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.716280    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.716412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.716305    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzxr\" (UniqueName: \"kubernetes.io/projected/4b073d99-dda0-4c19-aa4c-99b181577916-kube-api-access-9nzxr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.716504 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.716458    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.716604 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:48:48.716584    5510 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 21:48:48.716680 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:48:48.716667    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source podName:4b073d99-dda0-4c19-aa4c-99b181577916 nodeName:}" failed. No retries permitted until 2026-10-30 21:48:49.216643224 +0000 UTC m=+338.200550096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-6dcc7" (UID: "4b073d99-dda0-4c19-aa4c-99b181577916") : configmap "usage-tenancy-proxy-source" not found
Oct 30 21:48:48.717016 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.716987    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.718704 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.718679    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b073d99-dda0-4c19-aa4c-99b181577916-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.738028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.738005    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:48:48.740491 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.740465    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4f65d"]
Oct 30 21:48:48.774657 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.774633    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzxr\" (UniqueName: \"kubernetes.io/projected/4b073d99-dda0-4c19-aa4c-99b181577916-kube-api-access-9nzxr\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:48.893688 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:48.893645    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:49.123331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.123261    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfabbb4-81b3-442c-b9a4-69e4206308ce" path="/var/lib/kubelet/pods/abfabbb4-81b3-442c-b9a4-69e4206308ce/volumes"
Oct 30 21:48:49.221126 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.221096    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:49.222153 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.222125    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4b073d99-dda0-4c19-aa4c-99b181577916-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6dcc7\" (UID: \"4b073d99-dda0-4c19-aa4c-99b181577916\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:49.401257 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.401181    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:48:49.772832 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.772797    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"]
Oct 30 21:48:49.781114 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:48:49.781087    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b073d99_dda0_4c19_aa4c_99b181577916.slice/crio-96e5cd28aba45e86969cb16a797388ea2f11f113fbfff8ecfa581399ce9fef0a WatchSource:0}: Error finding container 96e5cd28aba45e86969cb16a797388ea2f11f113fbfff8ecfa581399ce9fef0a: Status 404 returned error can't find the container with id 96e5cd28aba45e86969cb16a797388ea2f11f113fbfff8ecfa581399ce9fef0a
Oct 30 21:48:49.782996 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:49.782980    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:48:50.449968 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:50.449936    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7" event={"ID":"4b073d99-dda0-4c19-aa4c-99b181577916","Type":"ContainerStarted","Data":"96e5cd28aba45e86969cb16a797388ea2f11f113fbfff8ecfa581399ce9fef0a"}
Oct 30 21:48:53.573307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:53.573262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.724490 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:53.724425    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:53.926532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:53.926367    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.079285 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:54.079249    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.238546 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:54.238484    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.388572 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:54.388534    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.554804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:54.554706    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:54.789211 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:48:54.789177    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:02.511208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:02.511169    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7" event={"ID":"4b073d99-dda0-4c19-aa4c-99b181577916","Type":"ContainerStarted","Data":"dd7d990cca48f2cfadd6fb914c368f26477377131f516e09d69dacbeda9224ab"}
Oct 30 21:49:02.511614 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:02.511248    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:49:02.618072 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:02.618015    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7" podStartSLOduration=2.270519062 podStartE2EDuration="14.618001725s" totalImagesPullingTime="12.347482663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:02.617424034 +0000 UTC m=+351.601330910" watchObservedRunningTime="2026-10-30 21:49:02.618001725 +0000 UTC m=+351.601908595"
Oct 30 21:49:13.278317 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:49:13.278282    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:49:13.521023 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:13.520992    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6dcc7"
Oct 30 21:49:15.820986 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:15.820937    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:16.039616 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:16.039580    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:17.822852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:17.822805    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:17.983693 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:17.983649    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:18.219130 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:18.219094    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:18.630417 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:18.630297    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:24.175652 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:24.175576    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:49:31.703523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.703487    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 21:49:31.745523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.745493    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780063    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780095    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780129    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780170    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780227    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780276 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780274    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780291    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsdk\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780339    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.780451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.780358    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.840953 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.840922    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 21:49:31.874248 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.874223    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 21:49:31.881546 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881555    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881621 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881578    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881712 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881696    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881762    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881869 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881827    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881866    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.881926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882020 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.881930    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsdk\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882020 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.882004    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.882079    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882191 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.882173    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882298 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.882279    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.882445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.882415    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.884131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.884112    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.884268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.884249    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.940151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.940123    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:31.940925 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:31.940897    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsdk\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\") pod \"maas-default-gateway-openshift-default-694db55c87-mbkwh\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:32.057887 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:32.057855    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:32.476539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:32.476499    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 21:49:32.484644 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:49:32.484615    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d6928a_1ea2_4c08_9760_627e4124e280.slice/crio-6ecac4b684a29ad8c9ded64092aad54452bf8bbdd65be258ec8b4f9ee3a25426 WatchSource:0}: Error finding container 6ecac4b684a29ad8c9ded64092aad54452bf8bbdd65be258ec8b4f9ee3a25426: Status 404 returned error can't find the container with id 6ecac4b684a29ad8c9ded64092aad54452bf8bbdd65be258ec8b4f9ee3a25426
Oct 30 21:49:32.631470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:32.631431    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" event={"ID":"c0d6928a-1ea2-4c08-9760-627e4124e280","Type":"ContainerStarted","Data":"6ecac4b684a29ad8c9ded64092aad54452bf8bbdd65be258ec8b4f9ee3a25426"}
Oct 30 21:49:36.516244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:36.516210    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:49:36.516626 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:36.516291    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:49:36.516626 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:36.516340    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:49:36.650293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:36.650257    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" event={"ID":"c0d6928a-1ea2-4c08-9760-627e4124e280","Type":"ContainerStarted","Data":"b70faa79210f5ca5feee10b8e2a7048ee68f0ea048b512c24c30bc71ed203f94"}
Oct 30 21:49:36.789901 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:36.789799    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podStartSLOduration=1.760907776 podStartE2EDuration="5.789761881s" totalImagesPullingTime="4.028854105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.788747192 +0000 UTC m=+385.772654070" watchObservedRunningTime="2026-10-30 21:49:36.789761881 +0000 UTC m=+385.773668819"
Oct 30 21:49:37.076856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:37.074446    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:37.077002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:37.076914    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:37.077002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:37.076980    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:38.071804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:38.064158    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:38.071804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:38.064219    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:38.922924 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:38.922886    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:39.069809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:39.069126    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:39.069809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:39.069189    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:39.139396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:39.139361    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:40.076935 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:40.076901    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:40.077211 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:40.077182    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:41.071262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:41.071227    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:41.071870 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:41.071839    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:42.073210 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:42.073174    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 21:49:42.073886 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:42.073853    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 21:49:43.063467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:43.063435    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:43.063708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:43.063688    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:43.064393 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:43.064372    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 21:49:43.278416 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:49:43.278378    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:49:48.202968 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:48.202933    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:56.193038 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:56.193002    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:56.361327 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:49:56.361293    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:05.736810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:05.736761    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:13.278386 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:50:13.278339    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:50:18.855207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:18.855156    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:19.014228 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:19.014189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:37.497815 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:37.497763    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:37.661304 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:37.661265    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:43.281502 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:50:43.281463    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:50:57.954258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:57.954180    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"]
Oct 30 21:50:57.957333 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:57.957317    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.006508 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.006482    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:50:58.051939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.051913    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:50:58.052095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.051943    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:50:58.052095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.051980    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sm8jq\""
Oct 30 21:50:58.053055 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.053023    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"]
Oct 30 21:50:58.073096 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.073070    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:58.109442 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109409    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc74g\" (UniqueName: \"kubernetes.io/projected/a0d8ee34-d09b-4cc9-a870-78846d79306c-kube-api-access-nc74g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.109580 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109446    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.109580 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109522    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d8ee34-d09b-4cc9-a870-78846d79306c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.109673 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109586    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.109673 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109611    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.109673 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.109643    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211038 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211012    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211050    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211080    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211114    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc74g\" (UniqueName: \"kubernetes.io/projected/a0d8ee34-d09b-4cc9-a870-78846d79306c-kube-api-access-nc74g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211149    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d8ee34-d09b-4cc9-a870-78846d79306c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211523 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211491    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211614 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.211614 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.211564    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.213300 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.213279    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d8ee34-d09b-4cc9-a870-78846d79306c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.213567 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.213550    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d8ee34-d09b-4cc9-a870-78846d79306c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.264861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.264838    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc74g\" (UniqueName: \"kubernetes.io/projected/a0d8ee34-d09b-4cc9-a870-78846d79306c-kube-api-access-nc74g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h\" (UID: \"a0d8ee34-d09b-4cc9-a870-78846d79306c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.269680 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.269664    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:50:58.636530 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:58.636447    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"]
Oct 30 21:50:58.644673 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:50:58.644644    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8ee34_d09b_4cc9_a870_78846d79306c.slice/crio-e8506ed0686331ca1c5edadd5a514b4c77548ec1ab19a1566cf7dab5fc2a5f96 WatchSource:0}: Error finding container e8506ed0686331ca1c5edadd5a514b4c77548ec1ab19a1566cf7dab5fc2a5f96: Status 404 returned error can't find the container with id e8506ed0686331ca1c5edadd5a514b4c77548ec1ab19a1566cf7dab5fc2a5f96
Oct 30 21:50:59.011878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:50:59.011845    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h" event={"ID":"a0d8ee34-d09b-4cc9-a870-78846d79306c","Type":"ContainerStarted","Data":"e8506ed0686331ca1c5edadd5a514b4c77548ec1ab19a1566cf7dab5fc2a5f96"}
Oct 30 21:51:00.047896 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:00.047855    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:51:01.021311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:01.021213    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h" event={"ID":"a0d8ee34-d09b-4cc9-a870-78846d79306c","Type":"ContainerStarted","Data":"d1efdb6a703b496263e36058bbd9af1865a75c7e0c5be3b48d4ad4aefbc9df26"}
Oct 30 21:51:01.130117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:01.130064    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h" podStartSLOduration=2.056765862 podStartE2EDuration="4.130051491s" totalImagesPullingTime="2.073285629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:01.129351588 +0000 UTC m=+470.113258463" watchObservedRunningTime="2026-10-30 21:51:01.130051491 +0000 UTC m=+470.113958366"
Oct 30 21:51:01.270905 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:01.270870    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:51:01.897575 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:01.897536    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:05.334657 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.334618    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"]
Oct 30 21:51:05.338183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.338165    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.370843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.370809    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.371028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.370863    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.371028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.370969    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cafbb56-06c8-47ab-9ec6-d5355209b352-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.371156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.371046    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.371156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.371101    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgqr\" (UniqueName: \"kubernetes.io/projected/7cafbb56-06c8-47ab-9ec6-d5355209b352-kube-api-access-xhgqr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.371258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.371159    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.432364 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.432320    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:51:05.433359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.433338    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"]
Oct 30 21:51:05.472202 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472164    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cafbb56-06c8-47ab-9ec6-d5355209b352-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472220    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472257    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgqr\" (UniqueName: \"kubernetes.io/projected/7cafbb56-06c8-47ab-9ec6-d5355209b352-kube-api-access-xhgqr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472275    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472300    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472339    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472776 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472751    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472776 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472768    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.472921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.472860    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.474542 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.474526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cafbb56-06c8-47ab-9ec6-d5355209b352-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.474821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.474802    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cafbb56-06c8-47ab-9ec6-d5355209b352-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.491571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.491538    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:05.526592 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.526564    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgqr\" (UniqueName: \"kubernetes.io/projected/7cafbb56-06c8-47ab-9ec6-d5355209b352-kube-api-access-xhgqr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr\" (UID: \"7cafbb56-06c8-47ab-9ec6-d5355209b352\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:05.651201 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:05.651116    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:06.038724 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:06.038688    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"]
Oct 30 21:51:06.046441 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:06.046412    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cafbb56_06c8_47ab_9ec6_d5355209b352.slice/crio-0cc539158b0450cccaad1bfde876e6259b73345a21a6c64f4494e95b1cb9264b WatchSource:0}: Error finding container 0cc539158b0450cccaad1bfde876e6259b73345a21a6c64f4494e95b1cb9264b: Status 404 returned error can't find the container with id 0cc539158b0450cccaad1bfde876e6259b73345a21a6c64f4494e95b1cb9264b
Oct 30 21:51:07.047538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:07.047504    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr" event={"ID":"7cafbb56-06c8-47ab-9ec6-d5355209b352","Type":"ContainerStarted","Data":"c09ce2fbc5af429ee56bcdeb3d5a8808abc32e8f86c9c64f3a0db4292d15e4b4"}
Oct 30 21:51:07.047959 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:07.047557    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr" event={"ID":"7cafbb56-06c8-47ab-9ec6-d5355209b352","Type":"ContainerStarted","Data":"0cc539158b0450cccaad1bfde876e6259b73345a21a6c64f4494e95b1cb9264b"}
Oct 30 21:51:07.155215 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:07.155160    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr" podStartSLOduration=1.5437500320000002 podStartE2EDuration="2.155144361s" totalImagesPullingTime="611.394329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:07.155132611 +0000 UTC m=+476.139039487" watchObservedRunningTime="2026-10-30 21:51:07.155144361 +0000 UTC m=+476.139051244"
Oct 30 21:51:07.651209 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:07.651169    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:08.270644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:08.270600    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:51:08.283254 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:08.283226    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:51:09.066695 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:09.066668    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h"
Oct 30 21:51:09.786227 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:09.786196    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:51:10.020500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:10.020469    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:13.278478 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:51:13.278440    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:51:15.651854 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:15.651817    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:15.664191 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:15.664164    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:16.041432 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:16.041401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:16.096980 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:16.096951    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr"
Oct 30 21:51:17.549705 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.549664    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 21:51:17.556539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.556515    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.647130 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.647098    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 21:51:17.678991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.678955    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679063    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679162    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679218    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679324 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679297    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679381 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679381 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679362    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xb5\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679609 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679399    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.679609 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.679429    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780062 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780033    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780072    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780109    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780134    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780173    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780198    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xb5\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780525 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780228    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780525 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780253    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780736 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780708    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780759    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780816    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.780918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780886    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.781010 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.780991    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.782904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.782880    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.783117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.783101    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.834126 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.834078    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xb5\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.834572 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.834551    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wz9xp\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:17.869678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:17.869643    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:18.299501 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:18.299468    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 21:51:18.338992 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:18.338954    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5bd04a_318a_4316_9f9f_425fd4aaca05.slice/crio-47be1c9a222e64d223bcd02d5695c53ddcfef95a070dcaa8923b5175de4b2ece WatchSource:0}: Error finding container 47be1c9a222e64d223bcd02d5695c53ddcfef95a070dcaa8923b5175de4b2ece: Status 404 returned error can't find the container with id 47be1c9a222e64d223bcd02d5695c53ddcfef95a070dcaa8923b5175de4b2ece
Oct 30 21:51:18.341630 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:18.341588    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:18.341713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:18.341683    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:18.341756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:18.341728    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:18.587616 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:18.587527    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:51:19.099448 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.099414    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" event={"ID":"3d5bd04a-318a-4316-9f9f-425fd4aaca05","Type":"ContainerStarted","Data":"a471c7d92fda8fe394237828fa2afe9f1bc874dbc65e360a2bf8376dce9ad636"}
Oct 30 21:51:19.099625 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.099476    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" event={"ID":"3d5bd04a-318a-4316-9f9f-425fd4aaca05","Type":"ContainerStarted","Data":"47be1c9a222e64d223bcd02d5695c53ddcfef95a070dcaa8923b5175de4b2ece"}
Oct 30 21:51:19.212976 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.212909    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podStartSLOduration=2.212891138 podStartE2EDuration="2.212891138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:19.211595737 +0000 UTC m=+488.195502637" watchObservedRunningTime="2026-10-30 21:51:19.212891138 +0000 UTC m=+488.196798015"
Oct 30 21:51:19.305334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.305300    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:19.569537 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.569427    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:19.872853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.872466    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:19.872853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.872592    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:19.872853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:19.872648    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:20.870819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:20.870393    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:20.870819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:20.870469    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:21.874805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:21.871193    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:21.874805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:21.871291    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:22.872176 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:22.871712    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:22.872176 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:22.871793    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:23.583800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.583252    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"]
Oct 30 21:51:23.587100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.587071    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.636534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.636508    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 21:51:23.644059 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644026    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7452z\" (UniqueName: \"kubernetes.io/projected/48e0d4a0-2d4f-4769-964f-29120855dee9-kube-api-access-7452z\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.644189 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644076    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.644290 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644213    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e0d4a0-2d4f-4769-964f-29120855dee9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.644290 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644277    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.644396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644312    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.644569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.644544    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.669682 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.669608    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:23.683265 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.683237    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"]
Oct 30 21:51:23.745334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745304    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745354    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7452z\" (UniqueName: \"kubernetes.io/projected/48e0d4a0-2d4f-4769-964f-29120855dee9-kube-api-access-7452z\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745385    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745427    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e0d4a0-2d4f-4769-964f-29120855dee9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745463    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745486    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.745928 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745901    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.746031 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745929    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.746031 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.745983    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.747883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.747863    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e0d4a0-2d4f-4769-964f-29120855dee9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.748110 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.748090    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e0d4a0-2d4f-4769-964f-29120855dee9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.799092 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.799067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7452z\" (UniqueName: \"kubernetes.io/projected/48e0d4a0-2d4f-4769-964f-29120855dee9-kube-api-access-7452z\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7fxjz\" (UID: \"48e0d4a0-2d4f-4769-964f-29120855dee9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:23.873864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.873159    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:23.873864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.873235    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:23.902711 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:23.902667    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:24.295010 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:24.294974    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"]
Oct 30 21:51:24.302212 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:24.302187    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e0d4a0_2d4f_4769_964f_29120855dee9.slice/crio-9b881cda812a89584c83be8eb00be5fe766cfeb4218c182cc70473e9627d0cdf WatchSource:0}: Error finding container 9b881cda812a89584c83be8eb00be5fe766cfeb4218c182cc70473e9627d0cdf: Status 404 returned error can't find the container with id 9b881cda812a89584c83be8eb00be5fe766cfeb4218c182cc70473e9627d0cdf
Oct 30 21:51:24.870414 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:24.870380    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 21:51:24.870864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:24.870430    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 21:51:25.127356 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.127272    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz" event={"ID":"48e0d4a0-2d4f-4769-964f-29120855dee9","Type":"ContainerStarted","Data":"270d67f02d09437db8866ad86c231702d853d193fec7e21e9ce6670e5e2dadcf"}
Oct 30 21:51:25.127356 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.127320    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz" event={"ID":"48e0d4a0-2d4f-4769-964f-29120855dee9","Type":"ContainerStarted","Data":"9b881cda812a89584c83be8eb00be5fe766cfeb4218c182cc70473e9627d0cdf"}
Oct 30 21:51:25.236238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.236178    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz" podStartSLOduration=1.636875297 podStartE2EDuration="2.236163195s" totalImagesPullingTime="599.287898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:25.235443121 +0000 UTC m=+494.219349997" watchObservedRunningTime="2026-10-30 21:51:25.236163195 +0000 UTC m=+494.220070064"
Oct 30 21:51:25.874921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.874891    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:25.875316 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.875170    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:25.875859 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.875840    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 21:51:25.902941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:25.902921    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:26.608709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.608674    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"]
Oct 30 21:51:26.612309 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.612287    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.661705 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.661674    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 21:51:26.667187 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.667152    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"]
Oct 30 21:51:26.671159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671138    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.671252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671175    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rg27\" (UniqueName: \"kubernetes.io/projected/4f613d67-925e-409b-9208-13ecbb1163a9-kube-api-access-4rg27\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.671252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671202    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.671355 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671338    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f613d67-925e-409b-9208-13ecbb1163a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.671463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671420    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.671463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.671460    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.727339 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.727309    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.772523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f613d67-925e-409b-9208-13ecbb1163a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.772619    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.772692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.773009    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.773078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rg27\" (UniqueName: \"kubernetes.io/projected/4f613d67-925e-409b-9208-13ecbb1163a9-kube-api-access-4rg27\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.773200    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.773802    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.774133    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.774347    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.779843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.778741    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f613d67-925e-409b-9208-13ecbb1163a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.780310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.779931    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f613d67-925e-409b-9208-13ecbb1163a9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.828165 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.828131    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rg27\" (UniqueName: \"kubernetes.io/projected/4f613d67-925e-409b-9208-13ecbb1163a9-kube-api-access-4rg27\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t8b8g\" (UID: \"4f613d67-925e-409b-9208-13ecbb1163a9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:26.925166 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:26.925078    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:27.310999 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:27.310965    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"]
Oct 30 21:51:27.320111 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:27.320078    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f613d67_925e_409b_9208_13ecbb1163a9.slice/crio-a6906aad6f0a8229b30f23ebf5a533d077b61da942b79cfad47c6069995ae8c5 WatchSource:0}: Error finding container a6906aad6f0a8229b30f23ebf5a533d077b61da942b79cfad47c6069995ae8c5: Status 404 returned error can't find the container with id a6906aad6f0a8229b30f23ebf5a533d077b61da942b79cfad47c6069995ae8c5
Oct 30 21:51:28.015540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.015502    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:28.142335 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.142297    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g" event={"ID":"4f613d67-925e-409b-9208-13ecbb1163a9","Type":"ContainerStarted","Data":"46b99a9771ada0786a003d184966787bba1d15088a22a32fae16270741361322"}
Oct 30 21:51:28.142475 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.142348    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g" event={"ID":"4f613d67-925e-409b-9208-13ecbb1163a9","Type":"ContainerStarted","Data":"a6906aad6f0a8229b30f23ebf5a533d077b61da942b79cfad47c6069995ae8c5"}
Oct 30 21:51:28.180853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.180077    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:28.280241 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.280167    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g" podStartSLOduration=1.678640316 podStartE2EDuration="2.280146627s" totalImagesPullingTime="601.506311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:28.273082238 +0000 UTC m=+497.256989115" watchObservedRunningTime="2026-10-30 21:51:28.280146627 +0000 UTC m=+497.264053506"
Oct 30 21:51:28.925539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:28.925510    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:32.689790 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.689755    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 21:51:32.693450 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.693431    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855470    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855507    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855525    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqjw\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855646    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855706    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855758    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855842    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855874    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.855961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.855895    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.877458 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.877425    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 21:51:32.957064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.956965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957018    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957285 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957131    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqjw\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957285 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957184    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957285 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957263    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957312    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957351    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957391    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957401    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957424    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957669 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957432    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957722 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957683    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.957799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.957722    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.958085 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.958063    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.959644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.959613    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:32.959737 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:32.959672    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:33.012362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.012333    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:33.012458 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.012369    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqjw\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\") pod \"maas-default-gateway-openshift-default-694db55c87-v4jj6\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:33.306130 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.306061    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:33.727837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.727798    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 21:51:33.734942 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:33.734905    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27964505_3e9a_4182_b996_db510769ff08.slice/crio-d33739c4ef73a3995999ce09159671c48927aabc74e2321980895fd24143e1f8 WatchSource:0}: Error finding container d33739c4ef73a3995999ce09159671c48927aabc74e2321980895fd24143e1f8: Status 404 returned error can't find the container with id d33739c4ef73a3995999ce09159671c48927aabc74e2321980895fd24143e1f8
Oct 30 21:51:33.737175 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.737143    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:33.737250 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.737213    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:33.737250 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.737241    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:51:33.903409 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.903371    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:33.915463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:33.915440    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:34.171350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.171259    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" event={"ID":"27964505-3e9a-4182-b996-db510769ff08","Type":"ContainerStarted","Data":"860aedf10bead954eaa331f67d4ce6144fd8d00e233b41084e9b02920acebaa5"}
Oct 30 21:51:34.171350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.171328    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" event={"ID":"27964505-3e9a-4182-b996-db510769ff08","Type":"ContainerStarted","Data":"d33739c4ef73a3995999ce09159671c48927aabc74e2321980895fd24143e1f8"}
Oct 30 21:51:34.183054 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.183016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7fxjz"
Oct 30 21:51:34.291095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.291034    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podStartSLOduration=2.291019863 podStartE2EDuration="2.291019863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.289510157 +0000 UTC m=+503.273417056" watchObservedRunningTime="2026-10-30 21:51:34.291019863 +0000 UTC m=+503.274926739"
Oct 30 21:51:34.307191 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.307158    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:34.308923 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.308898    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:34.308988 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.308955    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:34.631252 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:34.631213    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:51:35.307307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:35.307271    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:35.307856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:35.307320    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:36.313056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:36.313018    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:36.313925 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:36.313882    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:36.928076 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:36.926109    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:36.939526 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:36.939319    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:37.223901 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:37.223870    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g"
Oct 30 21:51:37.307294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:37.307260    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:37.307600 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:37.307550    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:38.307294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:38.307260    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:38.307673 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:38.307319    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:39.306536 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:39.306495    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:39.306729 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:39.306561    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:40.307293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:40.307253    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:51:40.307667 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:40.307316    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:51:42.307166 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:42.307120    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:42.307584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:42.307194    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:43.283201 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:51:43.283159    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:51:43.347928 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:43.347898    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:51:43.348371 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:43.347963    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:51:43.711125 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:43.711097    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:43.711522 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:43.711498    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:43.712235 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:43.712215    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 21:51:45.778013 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.777978    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:51:45.781486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.781458    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.785222 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.785198    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:45.838846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.838095    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:51:45.838846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.838616    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.838846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.838725    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn72d\" (UniqueName: \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.883970 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.883940    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:51:45.884117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.883940    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-h6t7v\""
Oct 30 21:51:45.885231 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.885211    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:51:45.939451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.939426    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.939601 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.939483    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn72d\" (UniqueName: \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.941987 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.941961    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:45.992583 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:45.992557    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn72d\" (UniqueName: \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\") pod \"ai-gateway-controller-7d6c696b49-8vvq9\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:46.057010 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:46.056929    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:46.095192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:46.095159    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:46.464507 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:46.464481    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:51:46.471893 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:46.471865    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb5d2b3_8d9b_44a4_a248_7d75552f64e0.slice/crio-fe6255a262bf5a05b27b8b03e209aba4b9846b39092999d8447f5fe7a83ee542 WatchSource:0}: Error finding container fe6255a262bf5a05b27b8b03e209aba4b9846b39092999d8447f5fe7a83ee542: Status 404 returned error can't find the container with id fe6255a262bf5a05b27b8b03e209aba4b9846b39092999d8447f5fe7a83ee542
Oct 30 21:51:47.254539 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:47.254507    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" event={"ID":"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0","Type":"ContainerStarted","Data":"fe6255a262bf5a05b27b8b03e209aba4b9846b39092999d8447f5fe7a83ee542"}
Oct 30 21:51:53.285532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:53.285489    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" event={"ID":"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0","Type":"ContainerStarted","Data":"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"}
Oct 30 21:51:53.286046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:53.285607    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:51:53.396032 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:53.395976    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" podStartSLOduration=1.953552631 podStartE2EDuration="8.395963284s" totalImagesPullingTime="6.442410653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:53.395578711 +0000 UTC m=+522.379485585" watchObservedRunningTime="2026-10-30 21:51:53.395963284 +0000 UTC m=+522.379870177"
Oct 30 21:51:55.611607 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:55.611559    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:55.998790 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:55.998735    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"]
Oct 30 21:51:56.002378 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.002354    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.066625 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.054599    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:51:56.105294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.105260    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-26prj\""
Oct 30 21:51:56.105520 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.105285    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"]
Oct 30 21:51:56.144056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.144024    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9798351-651c-4c76-a42e-06819a31734e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.144207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.144073    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9svn\" (UniqueName: \"kubernetes.io/projected/e9798351-651c-4c76-a42e-06819a31734e-kube-api-access-w9svn\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.245532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.245497    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9798351-651c-4c76-a42e-06819a31734e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.245721 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.245566    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9svn\" (UniqueName: \"kubernetes.io/projected/e9798351-651c-4c76-a42e-06819a31734e-kube-api-access-w9svn\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.246357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.246334    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9798351-651c-4c76-a42e-06819a31734e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.299454 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.299383    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9svn\" (UniqueName: \"kubernetes.io/projected/e9798351-651c-4c76-a42e-06819a31734e-kube-api-access-w9svn\") pod \"payload-pre-processing-7b98bb678c-xpk68\" (UID: \"e9798351-651c-4c76-a42e-06819a31734e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.315251 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.315226    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:51:56.680724 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.680648    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"]
Oct 30 21:51:56.688751 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:51:56.688721    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9798351_651c_4c76_a42e_06819a31734e.slice/crio-d2f2741d72eed321aa1f79d2b262a357b7d6189ad3cb1c65866192c0052aa706 WatchSource:0}: Error finding container d2f2741d72eed321aa1f79d2b262a357b7d6189ad3cb1c65866192c0052aa706: Status 404 returned error can't find the container with id d2f2741d72eed321aa1f79d2b262a357b7d6189ad3cb1c65866192c0052aa706
Oct 30 21:51:56.849602 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:56.849564    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:57.305767 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:51:57.305734    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68" event={"ID":"e9798351-651c-4c76-a42e-06819a31734e","Type":"ContainerStarted","Data":"d2f2741d72eed321aa1f79d2b262a357b7d6189ad3cb1c65866192c0052aa706"}
Oct 30 21:52:00.322047 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:00.322014    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68" event={"ID":"e9798351-651c-4c76-a42e-06819a31734e","Type":"ContainerStarted","Data":"df43a0d8ad6e308c1c25bf24d11a72fcdd169412df9b36eb10254ab49cf17df8"}
Oct 30 21:52:00.322469 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:00.322123    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:52:00.431770 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:00.431723    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68" podStartSLOduration=2.242585974 podStartE2EDuration="5.431710589s" totalImagesPullingTime="3.189124615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:00.431067007 +0000 UTC m=+529.414973883" watchObservedRunningTime="2026-10-30 21:52:00.431710589 +0000 UTC m=+529.415617465"
Oct 30 21:52:04.293191 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.293160    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:52:04.640992 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.640919    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"]
Oct 30 21:52:04.644637 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.644614    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.698361 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.698328    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"]
Oct 30 21:52:04.721480 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.721437    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7msv\" (UniqueName: \"kubernetes.io/projected/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-kube-api-access-t7msv\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.721643 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.721552    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-metrics-certs\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.822433 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.822394    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7msv\" (UniqueName: \"kubernetes.io/projected/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-kube-api-access-t7msv\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.822614 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.822447    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-metrics-certs\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.824861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.824842    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-metrics-certs\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.875203 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.875167    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7msv\" (UniqueName: \"kubernetes.io/projected/ebaabad6-8b43-4e0c-88c5-c34466bbe17f-kube-api-access-t7msv\") pod \"ai-gateway-controller-6678fd9b7b-bvmtm\" (UID: \"ebaabad6-8b43-4e0c-88c5-c34466bbe17f\") " pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:04.957329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:04.957249    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:05.322368 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:05.322330    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"]
Oct 30 21:52:05.329947 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:05.329921    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaabad6_8b43_4e0c_88c5_c34466bbe17f.slice/crio-2e370e3b207ac20733adebfb7ae22e4d2da15d0a47439d9c5e910e7b8d1eec9a WatchSource:0}: Error finding container 2e370e3b207ac20733adebfb7ae22e4d2da15d0a47439d9c5e910e7b8d1eec9a: Status 404 returned error can't find the container with id 2e370e3b207ac20733adebfb7ae22e4d2da15d0a47439d9c5e910e7b8d1eec9a
Oct 30 21:52:05.350229 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:05.350196    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm" event={"ID":"ebaabad6-8b43-4e0c-88c5-c34466bbe17f","Type":"ContainerStarted","Data":"2e370e3b207ac20733adebfb7ae22e4d2da15d0a47439d9c5e910e7b8d1eec9a"}
Oct 30 21:52:06.355918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:06.355877    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm" event={"ID":"ebaabad6-8b43-4e0c-88c5-c34466bbe17f","Type":"ContainerStarted","Data":"1844c31b6d45a57df90bb2b2b60f2449515987c14ec64cfdc075ede84e731385"}
Oct 30 21:52:06.356322 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:06.356011    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:06.464863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:06.464814    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm" podStartSLOduration=1.607955242 podStartE2EDuration="2.464800007s" totalImagesPullingTime="856.844765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:06.463369465 +0000 UTC m=+535.447276341" watchObservedRunningTime="2026-10-30 21:52:06.464800007 +0000 UTC m=+535.448706873"
Oct 30 21:52:11.330559 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:11.330533    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xpk68"
Oct 30 21:52:13.279093 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:52:13.279051    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:52:17.362950 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.362914    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6678fd9b7b-bvmtm"
Oct 30 21:52:17.516199 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.516170    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:52:17.516387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.516367    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" podUID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" containerName="manager" containerID="cri-o://bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b" gracePeriod=30
Oct 30 21:52:17.753527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.753502    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:52:17.841389 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.841361    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\") pod \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") "
Oct 30 21:52:17.841535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.841441    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\" (UniqueName: \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\") pod \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\" (UID: \"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0\") "
Oct 30 21:52:17.843527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.843489    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs" pod "ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" (UID: "ccb5d2b3-8d9b-44a4-a248-7d75552f64e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:17.843527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.843511    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d" pod "ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" (UID: "ccb5d2b3-8d9b-44a4-a248-7d75552f64e0"). InnerVolumeSpecName "kube-api-access-mn72d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:17.942288 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.942217    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:17.942288 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:17.942243    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn72d\" (UniqueName: \"kubernetes.io/projected/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0-kube-api-access-mn72d\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:18.409589 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.409552    5510 generic.go:356] "Generic (PLEG): container finished" podID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" containerID="bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b" exitCode=0
Oct 30 21:52:18.410043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.409599    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"
Oct 30 21:52:18.410043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.409633    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" event={"ID":"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0","Type":"ContainerDied","Data":"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"}
Oct 30 21:52:18.410043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.409690    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-8vvq9" event={"ID":"ccb5d2b3-8d9b-44a4-a248-7d75552f64e0","Type":"ContainerDied","Data":"fe6255a262bf5a05b27b8b03e209aba4b9846b39092999d8447f5fe7a83ee542"}
Oct 30 21:52:18.410043 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.409713    5510 scope.go:122] "RemoveContainer" containerID="bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"
Oct 30 21:52:18.418955 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.418932    5510 scope.go:122] "RemoveContainer" containerID="bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"
Oct 30 21:52:18.419214 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:52:18.419194    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b\": container with ID starting with bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b not found: ID does not exist" containerID="bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"
Oct 30 21:52:18.419264 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.419222    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b"} err="failed to get container status \"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b\": rpc error: code = NotFound desc = could not find container \"bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b\": container with ID starting with bae5b43c6746ff8111e4384de226d1e1ee920e3ccf6c5e9d853d805b717f9c3b not found: ID does not exist"
Oct 30 21:52:18.560571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.560542    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:52:18.606678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:18.606641    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-8vvq9"]
Oct 30 21:52:19.122802 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:19.122754    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" path="/var/lib/kubelet/pods/ccb5d2b3-8d9b-44a4-a248-7d75552f64e0/volumes"
Oct 30 21:52:19.261470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:19.261440    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:19.659164 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:19.659130    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:19.829417 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:19.829384    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:21.485366 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:21.485280    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:21.665176 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:21.665145    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:23.157069 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:23.157042    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:23.435447 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:23.435365    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:43.278593 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:52:43.278561    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:52:48.336462 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.336426    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:48.609483 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.609410    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:48.820608 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.820574    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 21:52:48.821010 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.820997    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" containerName="manager"
Oct 30 21:52:48.821063 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.821011    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" containerName="manager"
Oct 30 21:52:48.821103 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.821082    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb5d2b3-8d9b-44a4-a248-7d75552f64e0" containerName="manager"
Oct 30 21:52:48.824709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.824679    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.875796 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.875720    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5402dd-openshift-default-dockercfg-lstcs\""
Oct 30 21:52:48.918846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.918810    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.918905    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.918941    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919008    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919058    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919076    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfnh\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919193    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919234    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.919310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.919253    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:48.923420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.923386    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 21:52:48.992982 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:48.992952    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:49.019883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.019851    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.019883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.019888    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020048 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.019927    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020048 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.019962    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020048 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.019992    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020048 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020027    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020253 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020253 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020090    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfnh\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020253 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020136    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020407 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020324    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020762 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020762 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020618    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.020762 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020717    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.021012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.020889    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.022294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.022269    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.022365 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.022313    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.079241 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.079205    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.079979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.079879    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfnh\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.138472 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.138407    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:49.163754 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.163702    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 21:52:49.167810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.167767    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.216332 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.216304    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-14af5c-openshift-default-dockercfg-f5mqp\""
Oct 30 21:52:49.224322 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.224265    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 21:52:49.310793 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.310755    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:49.322537 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322511    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322652 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322545    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322652 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322562    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322725 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322696    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322759 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322743    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322833 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322817    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv8f\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322855    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322911 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.322911 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.322905    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424022    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv8f\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424056    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424079    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424198    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424268    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424292    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424309    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424348    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424407    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424443    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424516    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.424651 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424555    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.425023 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.424714    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.425023 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.425016    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.426463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.426446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.426603 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.426585    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.478376 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.478352    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv8f\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.494457 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.494437    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.523768 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.523739    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:49.576593 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.576566    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 21:52:49.583905 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:49.583875    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7311c13_cd2c_43e4_8dca_17fb4be77766.slice/crio-3981ee295a9e5da8bfb38a449c266e878f2f1dca3b5303b779f2d0fcb1e8ae4e WatchSource:0}: Error finding container 3981ee295a9e5da8bfb38a449c266e878f2f1dca3b5303b779f2d0fcb1e8ae4e: Status 404 returned error can't find the container with id 3981ee295a9e5da8bfb38a449c266e878f2f1dca3b5303b779f2d0fcb1e8ae4e
Oct 30 21:52:49.585934 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.585903    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:49.585998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.585964    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:49.585998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.585993    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:49.724456 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.724419    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:49.782772 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.782730    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:49.933579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:49.933543    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.098341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.098271    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.205162 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.205123    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 21:52:50.213605 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:50.213576    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eefeb76_ce93_44d6_a904_c60c75b6861c.slice/crio-1138672cbd3620c8e6b4d2b7a691b15ce36e516b1dfdec66c0a9378b0b1c5042 WatchSource:0}: Error finding container 1138672cbd3620c8e6b4d2b7a691b15ce36e516b1dfdec66c0a9378b0b1c5042: Status 404 returned error can't find the container with id 1138672cbd3620c8e6b4d2b7a691b15ce36e516b1dfdec66c0a9378b0b1c5042
Oct 30 21:52:50.215997 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.215966    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:50.216084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.216032    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:50.216084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.216061    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:50.220609 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.220586    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 21:52:50.226013 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.225985    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.294101 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.294068    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5be0d1-openshift-default-dockercfg-f2nmw\""
Oct 30 21:52:50.297196 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.297163    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 21:52:50.313229 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.313179    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.332970 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.332942    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.332984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333088    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333156    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333217    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333374 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333374 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333365    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333567 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333538    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgbn\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.333682 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.333666    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.434593 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434515    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgbn\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.434593 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434560    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434689    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434736    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434856    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434903    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434922    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434949    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.434982    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435553 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.435125    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435553 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.435352    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435553 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.435379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.435813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.435565    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.437128 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.437098    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.437128 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.437112    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.481513 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.481450    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.487640 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.487610    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.487640 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.487633    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgbn\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.562887 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.562845    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerStarted","Data":"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"}
Oct 30 21:52:50.563064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.562941    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerStarted","Data":"1138672cbd3620c8e6b4d2b7a691b15ce36e516b1dfdec66c0a9378b0b1c5042"}
Oct 30 21:52:50.564684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.564660    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerStarted","Data":"5955e135514f7510d295f6584d565c704e8ed4ea1f0f5bcec3e768b325e509ac"}
Oct 30 21:52:50.564806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.564700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerStarted","Data":"3981ee295a9e5da8bfb38a449c266e878f2f1dca3b5303b779f2d0fcb1e8ae4e"}
Oct 30 21:52:50.581256 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.581224    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:50.661788 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.661748    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.680337 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.680284    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podStartSLOduration=1.680264266 podStartE2EDuration="1.680264266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:50.678858719 +0000 UTC m=+579.662765596" watchObservedRunningTime="2026-10-30 21:52:50.680264266 +0000 UTC m=+579.664171143"
Oct 30 21:52:50.783308 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.783276    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:50.788211 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.788167    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podStartSLOduration=2.788151005 podStartE2EDuration="2.788151005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:50.786211424 +0000 UTC m=+579.770118302" watchObservedRunningTime="2026-10-30 21:52:50.788151005 +0000 UTC m=+579.772057881"
Oct 30 21:52:50.831207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.831173    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:50.889323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.889287    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 21:52:50.894187 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.894166    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:50.949445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.949413    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 21:52:50.988308 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:50.988266    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.015702 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.015673    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a07620-openshift-default-dockercfg-p8t9r\""
Oct 30 21:52:51.029151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.029120    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 21:52:51.045513 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045648 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045528    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045648 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045562    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045648 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045591    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045648 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045615    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045648 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045640    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dztg\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045927 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045678    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045927 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045703    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.045927 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.045722    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.055661 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:51.055625    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc1feed_9b03_4f35_b899_181c7cc4c5f2.slice/crio-b0bbc6deb653b6a43e3035e156e6c54adbdc0b0ca72fe93c8f2caf85f83678c0 WatchSource:0}: Error finding container b0bbc6deb653b6a43e3035e156e6c54adbdc0b0ca72fe93c8f2caf85f83678c0: Status 404 returned error can't find the container with id b0bbc6deb653b6a43e3035e156e6c54adbdc0b0ca72fe93c8f2caf85f83678c0
Oct 30 21:52:51.062198 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.062159    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.062311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.062245    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.062311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.062290    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.139360 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.139336    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:51.142323 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.142277    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.145082 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.145060    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:51.146509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146484    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146597 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146525    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146664 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146633    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146676    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146748    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146959 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146816    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146959 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146855    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dztg\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.146959 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.146907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.147271 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.147210    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.147373 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.147328    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.147445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.147396    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.147445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.147400    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.147597 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.147575    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.149064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.149038    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.149436 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.149420    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.201103 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.201068    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dztg\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.201369 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.201346    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.206869 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.206841    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:51.326556 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.326460    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.571242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.571205    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerStarted","Data":"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"}
Oct 30 21:52:51.571696 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.571262    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerStarted","Data":"b0bbc6deb653b6a43e3035e156e6c54adbdc0b0ca72fe93c8f2caf85f83678c0"}
Oct 30 21:52:51.571804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.571751    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:51.572730 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.572707    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:52:51.580087 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.580033    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 21:52:51.581447 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.581433    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:51.587320 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:51.587297    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879e9b5_fece_49a0_a747_e56664ef2df4.slice/crio-e55521df84080c0e3a1cd11c0ebefa8a0182becc48250337949b077f93d21c7a WatchSource:0}: Error finding container e55521df84080c0e3a1cd11c0ebefa8a0182becc48250337949b077f93d21c7a: Status 404 returned error can't find the container with id e55521df84080c0e3a1cd11c0ebefa8a0182becc48250337949b077f93d21c7a
Oct 30 21:52:51.589631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.589594    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.589712 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.589665    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.589712 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.589698    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:52:51.681250 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.681186    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podStartSLOduration=1.6811665329999999 podStartE2EDuration="1.681166533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:51.680380864 +0000 UTC m=+580.664287740" watchObservedRunningTime="2026-10-30 21:52:51.681166533 +0000 UTC m=+580.665073416"
Oct 30 21:52:51.788180 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:51.788148    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:52.577456 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.577419    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerStarted","Data":"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"}
Oct 30 21:52:52.577896 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.577485    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerStarted","Data":"e55521df84080c0e3a1cd11c0ebefa8a0182becc48250337949b077f93d21c7a"}
Oct 30 21:52:52.578012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.577992    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:52.579095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.579076    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:52:52.587312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.587289    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:52.688479 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:52.688428    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podStartSLOduration=2.688410206 podStartE2EDuration="2.688410206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:52.687868613 +0000 UTC m=+581.671775490" watchObservedRunningTime="2026-10-30 21:52:52.688410206 +0000 UTC m=+581.672317082"
Oct 30 21:52:53.207817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.207761    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:53.212876 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.212849    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:53.584669 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.584582    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:53.584669 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.584634    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:53.585336 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.585312    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:52:53.585439 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.585383    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:52:53.883967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:53.883842    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:54.091911 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:54.091879    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:54.458486 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:54.458452    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:54.675444 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:54.675408    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:54.839454 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:54.839354    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:55.336863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:55.336830    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:55.530282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:55.530244    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:55.680384 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:55.680299    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:55.948040 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:55.947944    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:56.128450 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:56.128379    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:56.287708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:56.287660    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.114835 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.114800    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.307258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.307209    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.496179 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.496148    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 21:52:58.499882 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.499858    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.616805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.597726    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 21:52:58.627498 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.618926    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-14af5c\""
Oct 30 21:52:58.627751 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.619017    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-14af5c-dockercfg-q5p8b\""
Oct 30 21:52:58.628017 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.622827    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtfh\" (UniqueName: \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.628282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.628260    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.640932 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.640903    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.729929 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.729896    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtfh\" (UniqueName: \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.729942    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.730635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.730614    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.788417 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.788345    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtfh\" (UniqueName: \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\") pod \"payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.813891 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.813863    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:58.828105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:58.828081    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:59.188454 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.188376    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 21:52:59.204793 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:52:59.204752    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc528264e_85f1_4100_9cfb_36393cd6517b.slice/crio-7dc8e6105c297cc0cc8746a92808b820f39149d42ba0b1280ba5629a41822625 WatchSource:0}: Error finding container 7dc8e6105c297cc0cc8746a92808b820f39149d42ba0b1280ba5629a41822625: Status 404 returned error can't find the container with id 7dc8e6105c297cc0cc8746a92808b820f39149d42ba0b1280ba5629a41822625
Oct 30 21:52:59.638171 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.638138    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" event={"ID":"c528264e-85f1-4100-9cfb-36393cd6517b","Type":"ContainerStarted","Data":"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"}
Oct 30 21:52:59.638347 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.638191    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" event={"ID":"c528264e-85f1-4100-9cfb-36393cd6517b","Type":"ContainerStarted","Data":"7dc8e6105c297cc0cc8746a92808b820f39149d42ba0b1280ba5629a41822625"}
Oct 30 21:52:59.638347 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.638212    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:52:59.747682 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.747635    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" podStartSLOduration=1.747620245 podStartE2EDuration="1.747620245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:59.746323414 +0000 UTC m=+588.730230291" watchObservedRunningTime="2026-10-30 21:52:59.747620245 +0000 UTC m=+588.731527121"
Oct 30 21:52:59.781736 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.781705    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:59.982204 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:52:59.982165    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:00.138084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:00.138027    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:02.723789 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:02.723727    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:02.889650 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:02.889614    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:04.675797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:04.675748    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:04.840519 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:04.840488    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:05.004905 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.004875    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:05.169353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.169322    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:05.331496 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.331414    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:05.526400 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.526363    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 21:53:05.530110 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.530087    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.581756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.581675    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a07620\""
Oct 30 21:53:05.633561 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.633513    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 21:53:05.633755 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.633691    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a07620-dockercfg-nt88r\""
Oct 30 21:53:05.646743 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.634523    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.647534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.647503    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44lr\" (UniqueName: \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.678840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.678800    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 21:53:05.682339 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.682317    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.742399 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.742371    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 21:53:05.748993 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.748957    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.749167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.749015    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.749167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.749057    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9sz\" (UniqueName: \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.749167 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.749083    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44lr\" (UniqueName: \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.750229 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.750206    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.808081 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.808045    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44lr\" (UniqueName: \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\") pod \"payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.843046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.842947    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:05.850607 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.850577    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.850737 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.850621    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9sz\" (UniqueName: \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.851468 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.851437    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.907816 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.906317    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9sz\" (UniqueName: \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\") pod \"payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:05.996129 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:05.996096    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:06.225839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.225809    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 21:53:06.227642 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.227618    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:06.232532 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:53:06.232506    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdd14ec_7908_4321_848c_c51cf2f14e7f.slice/crio-b1e2e76554e3a0f5df5beb79d2339ce38caa9edd26141d500cb9c547a6d919ec WatchSource:0}: Error finding container b1e2e76554e3a0f5df5beb79d2339ce38caa9edd26141d500cb9c547a6d919ec: Status 404 returned error can't find the container with id b1e2e76554e3a0f5df5beb79d2339ce38caa9edd26141d500cb9c547a6d919ec
Oct 30 21:53:06.368337 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.368305    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 21:53:06.376028 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:53:06.375994    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127344d8_b678_4c8e_935d_36d812f108e5.slice/crio-27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933 WatchSource:0}: Error finding container 27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933: Status 404 returned error can't find the container with id 27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933
Oct 30 21:53:06.411095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.411057    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:06.649583 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.649496    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:06.673255 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.673218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" event={"ID":"127344d8-b678-4c8e-935d-36d812f108e5","Type":"ContainerStarted","Data":"60e40e13934f671aad5b57225db7c8a53e2e9794cde59ad61c26e160f0b7b095"}
Oct 30 21:53:06.673449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.673280    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" event={"ID":"127344d8-b678-4c8e-935d-36d812f108e5","Type":"ContainerStarted","Data":"27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933"}
Oct 30 21:53:06.673449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.673310    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:06.674794 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.674749    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" event={"ID":"bfdd14ec-7908-4321-848c-c51cf2f14e7f","Type":"ContainerStarted","Data":"248750efeba8640fba482b9302ab7ac4e812b1f64e5605466f798bf1e3905399"}
Oct 30 21:53:06.674907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.674800    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" event={"ID":"bfdd14ec-7908-4321-848c-c51cf2f14e7f","Type":"ContainerStarted","Data":"b1e2e76554e3a0f5df5beb79d2339ce38caa9edd26141d500cb9c547a6d919ec"}
Oct 30 21:53:06.674907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.674844    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:06.807516 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.807452    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" podStartSLOduration=1.8074314949999999 podStartE2EDuration="1.807431495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:06.806271504 +0000 UTC m=+595.790178379" watchObservedRunningTime="2026-10-30 21:53:06.807431495 +0000 UTC m=+595.791338371"
Oct 30 21:53:06.921592 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:06.921451    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" podStartSLOduration=1.921432553 podStartE2EDuration="1.921432553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:06.920159253 +0000 UTC m=+595.904066129" watchObservedRunningTime="2026-10-30 21:53:06.921432553 +0000 UTC m=+595.905339429"
Oct 30 21:53:07.595561 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:07.595513    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.787026 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:07.786981    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.984445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:07.984417    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:53:08.147706 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.147674    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 21:53:08.151545 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.151525    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.248522 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.248438    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f776cb\""
Oct 30 21:53:08.248522 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.248518    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f776cb-dockercfg-vf8v4\""
Oct 30 21:53:08.250040 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.250015    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 21:53:08.284244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.284211    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spb69\" (UniqueName: \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.284426 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.284252    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.385307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.385257    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spb69\" (UniqueName: \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.385496 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.385322    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.386073 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.386054    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.441675 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.441645    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spb69\" (UniqueName: \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\") pod \"payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.464042 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.464010    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:08.892279 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:53:08.890309    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a70268_c93c_4ef6_ae1c_bae0636c20f5.slice/crio-fcd6d086b7c4fef97657ae480e489f359f2d24b5af9a3bf12e28c816fdfe5eda WatchSource:0}: Error finding container fcd6d086b7c4fef97657ae480e489f359f2d24b5af9a3bf12e28c816fdfe5eda: Status 404 returned error can't find the container with id fcd6d086b7c4fef97657ae480e489f359f2d24b5af9a3bf12e28c816fdfe5eda
Oct 30 21:53:08.900930 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:08.899917    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 21:53:09.691943 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.691907    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" event={"ID":"a3a70268-c93c-4ef6-ae1c-bae0636c20f5","Type":"ContainerStarted","Data":"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"}
Oct 30 21:53:09.692353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.691968    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" event={"ID":"a3a70268-c93c-4ef6-ae1c-bae0636c20f5","Type":"ContainerStarted","Data":"fcd6d086b7c4fef97657ae480e489f359f2d24b5af9a3bf12e28c816fdfe5eda"}
Oct 30 21:53:09.692353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.691994    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:09.801806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.801742    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" podStartSLOduration=1.80172847 podStartE2EDuration="1.80172847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:09.800969406 +0000 UTC m=+598.784876281" watchObservedRunningTime="2026-10-30 21:53:09.80172847 +0000 UTC m=+598.785635383"
Oct 30 21:53:09.811551 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.811522    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:09.988500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:09.988465    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.187342 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.187304    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.364760 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.364671    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:10.400611 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.400576    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 21:53:10.404336 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.404315    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.457799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.457058    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2d1f91\""
Oct 30 21:53:10.457799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.457259    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2d1f91-dockercfg-vpslr\""
Oct 30 21:53:10.466826 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.461288    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 21:53:10.616734 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.616651    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xghl\" (UniqueName: \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.616876 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.616861    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.644812 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.644790    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 21:53:10.717505 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.717475    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xghl\" (UniqueName: \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.717958 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.717585    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.718351 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.718330    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:10.777803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:10.777505    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xghl\" (UniqueName: \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\") pod \"payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:11.017379 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.017344    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:11.206418 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.206392    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:53:11.208213 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.208183    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:53:11.434849 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.434752    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 21:53:11.442377 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:53:11.442351    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb902d615_4402_4008_95ee_b048ca9be731.slice/crio-f4f96cfa736658666f8c98b72da2468b910c83bae27767fa7fcb15eb35843ed2 WatchSource:0}: Error finding container f4f96cfa736658666f8c98b72da2468b910c83bae27767fa7fcb15eb35843ed2: Status 404 returned error can't find the container with id f4f96cfa736658666f8c98b72da2468b910c83bae27767fa7fcb15eb35843ed2
Oct 30 21:53:11.703787 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.703692    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" event={"ID":"b902d615-4402-4008-95ee-b048ca9be731","Type":"ContainerStarted","Data":"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"}
Oct 30 21:53:11.703787 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.703753    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" event={"ID":"b902d615-4402-4008-95ee-b048ca9be731","Type":"ContainerStarted","Data":"f4f96cfa736658666f8c98b72da2468b910c83bae27767fa7fcb15eb35843ed2"}
Oct 30 21:53:11.704019 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.703842    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:11.818104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:11.818040    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" podStartSLOduration=1.8180213859999999 podStartE2EDuration="1.818021386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:11.816884326 +0000 UTC m=+600.800791202" watchObservedRunningTime="2026-10-30 21:53:11.818021386 +0000 UTC m=+600.801928265"
Oct 30 21:53:13.297351 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:53:13.297323    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:53:14.779064 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:14.779016    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:14.949172 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:14.949139    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:15.230540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:15.230505    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.529811 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:16.529766    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.737906 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:16.737874    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:16.998878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:16.998845    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.240012 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.239946    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:17.322966 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.322880    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:53:17.433873 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.433838    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 21:53:17.438701 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.438678    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.512534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.512358    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.512534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.512505    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xdb\" (UniqueName: \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.535212 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.535174    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5be0d1-dockercfg-492wc\""
Oct 30 21:53:17.536949 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.536927    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 21:53:17.537058 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.537039    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5be0d1\""
Oct 30 21:53:17.636853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.615265    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xdb\" (UniqueName: \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.636853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.615695    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.636853 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.616632    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.674361 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.674331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xdb\" (UniqueName: \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\") pod \"payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:17.682720 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.682690    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 21:53:17.682885 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.682867    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 21:53:17.744531 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.744502    5510 generic.go:356] "Generic (PLEG): container finished" podID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerID="5955e135514f7510d295f6584d565c704e8ed4ea1f0f5bcec3e768b325e509ac" exitCode=0
Oct 30 21:53:17.744685 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.744534    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerDied","Data":"5955e135514f7510d295f6584d565c704e8ed4ea1f0f5bcec3e768b325e509ac"}
Oct 30 21:53:17.744885 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.744869    5510 scope.go:122] "RemoveContainer" containerID="5955e135514f7510d295f6584d565c704e8ed4ea1f0f5bcec3e768b325e509ac"
Oct 30 21:53:17.746110 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.746080    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:17.746206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.746147    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:17.746206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.746184    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:17.793562 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:17.793542    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:18.212801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.212720    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 21:53:18.779804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.779016    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerStarted","Data":"98fd0325e12c46c7120404d25a8b619419fadeaa4c073aa2a9e81510bb4e268e"}
Oct 30 21:53:18.779804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.779135    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:18.779804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.779165    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" event={"ID":"67065ea8-6706-4f52-991f-b7153cf55f6e","Type":"ContainerStarted","Data":"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"}
Oct 30 21:53:18.779804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.779189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" event={"ID":"67065ea8-6706-4f52-991f-b7153cf55f6e","Type":"ContainerStarted","Data":"9e296a6a961dc28f7db6cd607cee8c92020db9c592681048a35d062b9289bbb1"}
Oct 30 21:53:18.911461 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:18.904994    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" podStartSLOduration=1.904974378 podStartE2EDuration="1.904974378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:18.904218719 +0000 UTC m=+607.888125596" watchObservedRunningTime="2026-10-30 21:53:18.904974378 +0000 UTC m=+607.888881260"
Oct 30 21:53:19.148858 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:19.143244    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 21:53:19.148858 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:19.143306    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 21:53:19.150510 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:19.150061    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:53:19.920920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:19.920886    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:20.123112 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:20.123077    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:20.139526 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:20.139498    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 21:53:20.139649 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:20.139542    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 21:53:20.702825 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:20.702763    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 21:53:21.139105 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:21.139072    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 21:53:21.139491 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:21.139124    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 21:53:21.581804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:21.577835    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:53:22.138768 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:22.138728    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 21:53:22.138956 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:22.138820    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 21:53:22.718703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:22.718671    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 21:53:23.046365 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:23.046290    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:23.139800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:23.139128    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 21:53:23.139800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:23.139180    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 21:53:24.145078 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:24.145004    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:53:24.145945 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:24.145923    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 21:53:24.628662 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:24.628628    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.797823 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:24.797766    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:24.996520 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:24.996488    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:26.994950 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:26.994916    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:53:27.850258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.850227    5510 generic.go:356] "Generic (PLEG): container finished" podID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerID="0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0" exitCode=0
Oct 30 21:53:27.850434 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.850283    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerDied","Data":"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"}
Oct 30 21:53:27.850652 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.850638    5510 scope.go:122] "RemoveContainer" containerID="0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"
Oct 30 21:53:27.851864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.851834    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:27.851969 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.851939    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:27.852031 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:27.851989    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:28.813162 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:28.813130    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.886818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:28.885580    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerStarted","Data":"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"}
Oct 30 21:53:28.981335 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:28.981287    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.207542 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.207450    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:53:29.207690 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.207602    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:53:29.207690 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.207661    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:53:29.210656 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.210631    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 21:53:29.213756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.213736    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.270806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.267970    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-17d809-dockercfg-w5scv\""
Oct 30 21:53:29.270806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.268082    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-17d809\""
Oct 30 21:53:29.271087 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.270987    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 21:53:29.326577 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.316469    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67f9z\" (UniqueName: \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.327093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.327061    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.431799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.428415    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67f9z\" (UniqueName: \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.431799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.428590    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.431799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.429577    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.484366 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.484308    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67f9z\" (UniqueName: \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\") pod \"payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.526318 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.526281    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:29.531143 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.531118    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.803291 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.803218    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 21:53:29.949926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:29.949895    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 21:53:30.207267 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:30.207185    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:53:30.207267 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:30.207241    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:53:30.897205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:30.897166    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" event={"ID":"5a619273-e44e-4876-bb4a-865f68eaa0f3","Type":"ContainerStarted","Data":"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"}
Oct 30 21:53:30.897383 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:30.897224    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" event={"ID":"5a619273-e44e-4876-bb4a-865f68eaa0f3","Type":"ContainerStarted","Data":"6599640e808141634c3853a139ea2996e2c37aa5800f3f4190146dcdcc365787"}
Oct 30 21:53:30.897383 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:30.897254    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:31.006046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:31.005971    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" podStartSLOduration=2.005950951 podStartE2EDuration="2.005950951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:31.004662219 +0000 UTC m=+619.988569095" watchObservedRunningTime="2026-10-30 21:53:31.005950951 +0000 UTC m=+619.989857829"
Oct 30 21:53:31.207341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:31.207250    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:53:31.207341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:31.207310    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:53:32.207504 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:32.207471    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:53:32.207902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:32.207524    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:53:33.207196 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:33.207154    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:53:33.207374 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:33.207211    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:53:33.976325 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:33.976270    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.125934 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:34.125905    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.223904 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:34.223353    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:53:34.224097 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:34.224057    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:53:34.228856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:34.225386    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 21:53:34.287583 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:34.287550    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.120967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:39.120938    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.309011 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:39.308972    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:39.865642 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:39.865606    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:40.016564 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:40.016494    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:40.184505 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:40.184477    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:41.905825 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:41.905768    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 21:53:42.152668 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:42.152625    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:53:43.310343 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:53:43.310247    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:53:43.681860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.680951    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 21:53:43.685498 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.685469    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.732686 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.732658    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 21:53:43.779027 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.778848    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"openshift-service-ca.crt\""
Oct 30 21:53:43.779027 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.778949    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"kube-root-ca.crt\""
Oct 30 21:53:43.779027 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.778991    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"simulator-w5-5402dd-kserve-self-signed-certs\""
Oct 30 21:53:43.779744 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.779719    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"default-dockercfg-9lrlz\""
Oct 30 21:53:43.784660 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.784638    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:43.860474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.860430    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.860725 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.860705    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.864801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.860925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.864801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.860973    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.864801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.861051    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.864801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.861078    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjfs\" (UniqueName: \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.964433 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.964393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.966726 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.966703    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.966985 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.966967    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.967112 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.967098    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.967274 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.967242    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.967386 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.967369    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjfs\" (UniqueName: \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.967854 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.966077    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:43.968856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.968548    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.968856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.966489    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.969345 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.969326    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.977112 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.977056    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.992953 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.992926    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:43.999346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.999314    5510 generic.go:356] "Generic (PLEG): container finished" podID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerID="7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e" exitCode=0
Oct 30 21:53:43.999489 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.999349    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerDied","Data":"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"}
Oct 30 21:53:43.999839 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:43.999766    5510 scope.go:122] "RemoveContainer" containerID="7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"
Oct 30 21:53:44.001312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.001280    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:44.001393 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.001376    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:44.001441 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.001424    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:44.025605 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.025583    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjfs\" (UniqueName: \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\") pod \"simulator-w5-5402dd-kserve-76d6d56c45-zsrfd\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") " pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:44.308367 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.308141    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:44.729663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:44.729632    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 21:53:44.738348 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:53:44.738317    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b45b54_e0d9_494c_9e1c_93dc13f87804.slice/crio-486aa00e08cf0a0ae787df69976c5dd87885244bacfd5a3469dacc290bc5131e WatchSource:0}: Error finding container 486aa00e08cf0a0ae787df69976c5dd87885244bacfd5a3469dacc290bc5131e: Status 404 returned error can't find the container with id 486aa00e08cf0a0ae787df69976c5dd87885244bacfd5a3469dacc290bc5131e
Oct 30 21:53:45.016863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.016059    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerStarted","Data":"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"}
Oct 30 21:53:45.018566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.018514    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" event={"ID":"85b45b54-e0d9-494c-9e1c-93dc13f87804","Type":"ContainerStarted","Data":"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"}
Oct 30 21:53:45.018566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.018549    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" event={"ID":"85b45b54-e0d9-494c-9e1c-93dc13f87804","Type":"ContainerStarted","Data":"486aa00e08cf0a0ae787df69976c5dd87885244bacfd5a3469dacc290bc5131e"}
Oct 30 21:53:45.303376 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.303245    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podStartSLOduration=2.303220558 podStartE2EDuration="2.303220558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:45.291596171 +0000 UTC m=+634.275503048" watchObservedRunningTime="2026-10-30 21:53:45.303220558 +0000 UTC m=+634.287127436"
Oct 30 21:53:45.309574 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.309538    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:45.786814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.786222    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:53:45.786814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.786412    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:53:45.786814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:45.786459    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 21:53:46.783805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:46.783186    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:53:46.783805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:46.783266    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 21:53:47.785215 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:47.785170    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:53:47.785842 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:47.785808    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 21:53:48.736286 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:48.734171    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:48.783806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:48.783163    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:53:48.783806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:48.783224    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 21:53:48.943458 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:48.943421    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:49.101635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:49.101557    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:49.783401 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:49.783354    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 21:53:49.783600 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:49.783415    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 21:53:50.218801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:50.211707    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:53:50.788092 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:50.788061    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:53:50.788476 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:50.788453    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:53:50.789247 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:50.789231    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 21:53:53.103318 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.103287    5510 generic.go:356] "Generic (PLEG): container finished" podID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerID="b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff" exitCode=0
Oct 30 21:53:53.103759 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.103355    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerDied","Data":"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"}
Oct 30 21:53:53.103759 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.103726    5510 scope.go:122] "RemoveContainer" containerID="b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"
Oct 30 21:53:53.105082 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.105055    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:53.105179 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.105119    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:53.105179 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.105149    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:53:53.585368 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.585329    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:53:53.657534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:53.657492    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:54.111039 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.110995    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerStarted","Data":"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"}
Oct 30 21:53:54.308711 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.308678    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:54.321746 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.321719    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:54.582160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.582121    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:53:54.583362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.583340    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:53:54.583449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:54.583388    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:53:55.131151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:55.131123    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 21:53:55.582342 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:55.582303    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:53:55.582590 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:55.582380    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:53:56.582351 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:56.582311    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:53:56.582740 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:56.582382    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:53:57.582312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:57.582268    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:53:57.582516 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:57.582328    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:53:58.582210 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:58.582169    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:53:58.582520 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:58.582489    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:53:58.623861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:58.623823    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:59.585678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:59.585647    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:53:59.586094 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:59.585983    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:53:59.586584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:53:59.586565    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 21:54:00.605115 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:00.605072    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:00.973990 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:00.973963    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.237812 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:01.237693    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.617270 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:01.617188    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.771584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:01.771503    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:02.075019 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:02.074983    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:03.448976 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:03.448940    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:08.256399 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:08.256357    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:08.415871 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:08.415838    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:09.571995 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:09.571969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.175771 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:10.175730    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.337469 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:10.337438    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:11.781498 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:11.781466    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:54:13.258887 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:13.258853    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:13.278740 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:54:13.278705    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:54:18.083544 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:18.083508    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:20.300909 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:20.300861    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:20.801769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:20.801733    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:21.190984 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:21.190903    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:21.342223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:21.342193    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:22.801541 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:22.801508    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:27.142501 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:27.142472    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:27.319512 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:27.319482    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.422584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:31.422552    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.570261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:31.570202    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.730988 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:31.730955    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:31.891569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:31.891534    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:33.817315 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:33.817233    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:54:35.936033 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:35.935998    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.551224 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:36.551193    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:36.723005 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:36.722975    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:37.622715 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.622685    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 21:54:37.626678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.626651    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.725394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.725363    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"premium-w5-5402dd-kserve-self-signed-certs\""
Oct 30 21:54:37.727147 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.727126    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 21:54:37.731083 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731053    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.731195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731090    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.731195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731124    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.731301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731280    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.731341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.731421 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.731407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdzf\" (UniqueName: \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.755642 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.755600    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:37.832957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.832922    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdzf\" (UniqueName: \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833145 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833030    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833145 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833064    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833145 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833098    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833201    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833507 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833479    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833507 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833499    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.833625 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.833538    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.835429 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.835402    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.835675 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.835657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.887357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.887299    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdzf\" (UniqueName: \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\") pod \"premium-w5-5402dd-kserve-79f596689b-q9xcl\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") " pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:37.939108 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:37.939083    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:38.320909 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:38.320875    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 21:54:38.327968 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:54:38.327940    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf3f0ff8_7975_458b_9af4_7e31beae3e52.slice/crio-bc2cdc140c1e15a4b737e5826adf1ff8590a14905dd817f31c24ca447754ec3b WatchSource:0}: Error finding container bc2cdc140c1e15a4b737e5826adf1ff8590a14905dd817f31c24ca447754ec3b: Status 404 returned error can't find the container with id bc2cdc140c1e15a4b737e5826adf1ff8590a14905dd817f31c24ca447754ec3b
Oct 30 21:54:38.349299 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:38.349274    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" event={"ID":"df3f0ff8-7975-458b-9af4-7e31beae3e52","Type":"ContainerStarted","Data":"bc2cdc140c1e15a4b737e5826adf1ff8590a14905dd817f31c24ca447754ec3b"}
Oct 30 21:54:39.356438 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:39.356399    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" event={"ID":"df3f0ff8-7975-458b-9af4-7e31beae3e52","Type":"ContainerStarted","Data":"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"}
Oct 30 21:54:39.463451 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:39.463399    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podStartSLOduration=2.463387836 podStartE2EDuration="2.463387836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:39.462617427 +0000 UTC m=+688.446524293" watchObservedRunningTime="2026-10-30 21:54:39.463387836 +0000 UTC m=+688.447294711"
Oct 30 21:54:39.939657 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:39.939621    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:41.007362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:41.007327    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:41.186953 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:41.186922    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:43.279919 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:54:43.279882    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:54:45.633710 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:45.633669    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:47.939571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:47.939535    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:47.952200 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:47.952175    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:48.423631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:48.423602    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 21:54:50.133186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:50.133143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:51.111341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.111308    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 21:54:51.115227 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.115202    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.212735 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.212698    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"openshift-service-ca.crt\""
Oct 30 21:54:51.212735 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.212712    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"kube-root-ca.crt\""
Oct 30 21:54:51.213151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.212826    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"default-dockercfg-xf7tg\""
Oct 30 21:54:51.213151 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.212939    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"simulator-w3-f776cb-kserve-self-signed-certs\""
Oct 30 21:54:51.214716 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.214692    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 21:54:51.215178 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.215157    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.277703    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.277856    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.277892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.277919    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvxq\" (UniqueName: \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.278100    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.279192 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.278175    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379298 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379217    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379298 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379262    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379371    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379412    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379431    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379448    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvxq\" (UniqueName: \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379718 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379612    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379836 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379815    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.379905 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.379878    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.381577 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.381552    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.382090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.382069    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.433891 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.433862    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvxq\" (UniqueName: \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\") pod \"simulator-w3-f776cb-kserve-c5d98f599-fmtk6\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") " pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:51.728489 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:51.728458    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:52.106002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:52.105923    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 21:54:52.113181 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:54:52.113150    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a278fdd_ff55_4c0f_9c6e_e64469fc241f.slice/crio-b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6 WatchSource:0}: Error finding container b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6: Status 404 returned error can't find the container with id b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6
Oct 30 21:54:52.437090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:52.437013    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" event={"ID":"6a278fdd-ff55-4c0f-9c6e-e64469fc241f","Type":"ContainerStarted","Data":"6014c6f85254931b0ae557a4545146610cc5773028369fe8704f0307200ad6fa"}
Oct 30 21:54:52.437090 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:52.437056    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" event={"ID":"6a278fdd-ff55-4c0f-9c6e-e64469fc241f","Type":"ContainerStarted","Data":"b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6"}
Oct 30 21:54:52.546327 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:52.546268    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podStartSLOduration=1.546253976 podStartE2EDuration="1.546253976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:52.544913505 +0000 UTC m=+701.528820380" watchObservedRunningTime="2026-10-30 21:54:52.546253976 +0000 UTC m=+701.530160851"
Oct 30 21:54:52.729293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:52.729262    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:54:54.601287 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:54.601247    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:56.845310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:56.845276    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:54:57.198230 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.198153    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 21:54:57.202004 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.201981    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237061 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237031    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237240 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237069    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237240 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237097    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237240 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237135    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237327    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgdn\" (UniqueName: \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.237408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.237370    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.252346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.252314    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"openshift-service-ca.crt\""
Oct 30 21:54:57.298161 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.298133    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"kube-root-ca.crt\""
Oct 30 21:54:57.298326 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.298308    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"default-dockercfg-mkbfj\""
Oct 30 21:54:57.298390 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.298308    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5be0d1\"/\"simulator-w2-5be0d1-kserve-self-signed-certs\""
Oct 30 21:54:57.300281 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.300256    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 21:54:57.321101 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.321065    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:57.338445 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338423    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgdn\" (UniqueName: \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338466    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338493    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338522    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338547    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338577    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338898 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338883    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.338963 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.338941    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.339037 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.339019    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.340833 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.340814    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.340944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.340928    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.404740 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.404691    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgdn\" (UniqueName: \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\") pod \"simulator-w2-5be0d1-kserve-6867554d6b-xxkzg\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") " pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.515830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.515772    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:57.891889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:57.891807    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 21:54:57.899109 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:54:57.899079    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf17584_09fb_41a8_a269_e165ed0048ed.slice/crio-6e38134085b0ccd95e1a3ac99b8a4a3a97c076b8d5b750aa03cec7a28a83d6d1 WatchSource:0}: Error finding container 6e38134085b0ccd95e1a3ac99b8a4a3a97c076b8d5b750aa03cec7a28a83d6d1: Status 404 returned error can't find the container with id 6e38134085b0ccd95e1a3ac99b8a4a3a97c076b8d5b750aa03cec7a28a83d6d1
Oct 30 21:54:58.471578 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:58.471536    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" event={"ID":"baf17584-09fb-41a8-a269-e165ed0048ed","Type":"ContainerStarted","Data":"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"}
Oct 30 21:54:58.471578 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:58.471579    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" event={"ID":"baf17584-09fb-41a8-a269-e165ed0048ed","Type":"ContainerStarted","Data":"6e38134085b0ccd95e1a3ac99b8a4a3a97c076b8d5b750aa03cec7a28a83d6d1"}
Oct 30 21:54:58.516476 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:58.516445    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:54:58.582159 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:58.582109    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podStartSLOduration=1.582097072 podStartE2EDuration="1.582097072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:58.581290055 +0000 UTC m=+707.565196932" watchObservedRunningTime="2026-10-30 21:54:58.582097072 +0000 UTC m=+707.566003948"
Oct 30 21:54:59.041396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:54:59.041356    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:01.728817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:01.728765    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:55:01.741419 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:01.741394    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:55:02.505095 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:02.505062    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 21:55:03.545450 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:03.545417    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:07.516178 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:07.516145    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:55:07.529388 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:07.529362    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:55:08.029821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:08.029765    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:08.528552 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:08.528521    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 21:55:12.523641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:12.523610    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:13.281014 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:55:13.280968    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:55:17.028478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:17.028447    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:18.812206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:18.812174    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:55:19.778069 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:19.778018    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:20.648818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:20.648774    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:21.729650 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:21.729618    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:23.902190 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:23.902156    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:24.236368 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:24.236332    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:26.114714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:26.114678    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:26.480645 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:26.480609    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:30.649615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:30.649509    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:35.180143 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:35.180050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:37.006633 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.006595    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 21:55:37.010473 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.010450    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.064797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.064752    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"openshift-service-ca.crt\""
Oct 30 21:55:37.097467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097439    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.097641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097480    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgck\" (UniqueName: \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.097717 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097643    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.097717 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097669    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.097717 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097691    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.097921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.097736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.106793 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.106750    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"kube-root-ca.crt\""
Oct 30 21:55:37.106922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.106835    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"premium-w1-a07620-kserve-self-signed-certs\""
Oct 30 21:55:37.106922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.106894    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a07620\"/\"default-dockercfg-4gr9b\""
Oct 30 21:55:37.112707 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.112677    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 21:55:37.136875 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.136845    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:37.198620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198585    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.198620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198624    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.198908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198647    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.198908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198670    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.198908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198695    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.198908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.198720    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgck\" (UniqueName: \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.199127 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.199073    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.199127 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.199089    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.199196 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.199123    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.200981 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.200956    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.201394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.201379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.253296 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.253270    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgck\" (UniqueName: \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\") pod \"premium-w1-a07620-kserve-6cdf8587d7-l2c2s\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") " pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.325725 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.325667    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:37.728001 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:37.727969    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 21:55:37.735135 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:55:37.735109    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c725fa_33cd_4ad1_97dd_4a89faa5c758.slice/crio-daf2f33a096cc68220982d6ef0bdeedac4fd5d3115c7a89d1a2ff69506cbe454 WatchSource:0}: Error finding container daf2f33a096cc68220982d6ef0bdeedac4fd5d3115c7a89d1a2ff69506cbe454: Status 404 returned error can't find the container with id daf2f33a096cc68220982d6ef0bdeedac4fd5d3115c7a89d1a2ff69506cbe454
Oct 30 21:55:38.733877 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:38.733841    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" event={"ID":"c1c725fa-33cd-4ad1-97dd-4a89faa5c758","Type":"ContainerStarted","Data":"7dd2b5504cc636fe48faa8633a4684e7ea85710f19c7e38e4e0c2b41a9097cd4"}
Oct 30 21:55:38.734260 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:38.733886    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" event={"ID":"c1c725fa-33cd-4ad1-97dd-4a89faa5c758","Type":"ContainerStarted","Data":"daf2f33a096cc68220982d6ef0bdeedac4fd5d3115c7a89d1a2ff69506cbe454"}
Oct 30 21:55:38.857677 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:38.857609    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podStartSLOduration=2.857591275 podStartE2EDuration="2.857591275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:38.854198035 +0000 UTC m=+747.838104912" watchObservedRunningTime="2026-10-30 21:55:38.857591275 +0000 UTC m=+747.841498155"
Oct 30 21:55:39.326610 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:39.326575    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:40.004224 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:40.004189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:43.279543 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:55:43.279511    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:55:44.445463 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:44.445402    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:45.144867 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:45.144836    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:55:47.326298 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:47.326268    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:47.338622 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:47.338595    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:47.801098 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:47.801066    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 21:55:48.999274 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:48.999238    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:52.138328 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.135744    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 21:55:52.142068 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.142042    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.191180 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.191154    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2d1f91-openshift-default-dockercfg-zn7vw\""
Oct 30 21:55:52.239134 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.239101    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 21:55:52.265071 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265039    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265220 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265106    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265324 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265303    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265410 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265392    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265482 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265466    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265533 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265502    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265626    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265683 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265664    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.265736 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.265723    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.366988 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.366947    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367144 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367007    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367144 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367072    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367144 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367113    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367144 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367140    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367202    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367242    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367483 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367483 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367423    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367483 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367456    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367637 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367596    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.367804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.367752    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.368068 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.368048    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.369384 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.369364    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.369584 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.369568    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.420154 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.420093    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.420384 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.420361    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.456238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.456213    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:52.881200 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.881167    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 21:55:52.887677 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:55:52.887648    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b740417_1017_43a0_840f_57ca770df230.slice/crio-94395091c7cb74b82ca6e3e2e6641cc72e4b0914727dab467b87879baa21da2e WatchSource:0}: Error finding container 94395091c7cb74b82ca6e3e2e6641cc72e4b0914727dab467b87879baa21da2e: Status 404 returned error can't find the container with id 94395091c7cb74b82ca6e3e2e6641cc72e4b0914727dab467b87879baa21da2e
Oct 30 21:55:52.890099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.890068    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:55:52.890177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.890135    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:55:52.890177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:52.890163    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:55:53.591373 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:53.591324    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:53.824965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:53.824926    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" event={"ID":"8b740417-1017-43a0-840f-57ca770df230","Type":"ContainerStarted","Data":"e06b01ffa3dce53a4620795d8dc3c493108cc98f993e531c54159259c360f3b2"}
Oct 30 21:55:53.825158 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:53.824982    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" event={"ID":"8b740417-1017-43a0-840f-57ca770df230","Type":"ContainerStarted","Data":"94395091c7cb74b82ca6e3e2e6641cc72e4b0914727dab467b87879baa21da2e"}
Oct 30 21:55:53.990808 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:53.990740    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podStartSLOduration=1.990724773 podStartE2EDuration="1.990724773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:53.988689344 +0000 UTC m=+762.972596223" watchObservedRunningTime="2026-10-30 21:55:53.990724773 +0000 UTC m=+762.974631649"
Oct 30 21:55:54.464861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:54.461599    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:54.464861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:54.461749    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:55:54.464861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:54.461825    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:55:55.459227 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:55.459190    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:55:55.459820 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:55.459762    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:55:56.461142 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:56.461109    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:55:56.461694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:56.461654    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:55:57.465804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:57.462161    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:55:57.465804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:57.462225    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:55:58.062371 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:58.062332    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:58.466184 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:58.466152    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:55:58.466746 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:58.466718    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:55:59.564394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:59.564355    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:59.565234 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:59.565206    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:55:59.566099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:55:59.566080    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 21:56:02.355131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:02.355093    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:02.646307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:02.646232    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:07.171413 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:07.171356    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:11.623264 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:11.623222    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:12.204461 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.204424    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"]
Oct 30 21:56:12.208846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.208821    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.266460 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.266423    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:12.313046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.313012    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:12.313263 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.313052    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:12.313263 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.313170    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:56:12.315216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.315193    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"]
Oct 30 21:56:12.317547 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.317524    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm2t\" (UniqueName: \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.317644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.317566    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.317700 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.317672    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.419193 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.419155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm2t\" (UniqueName: \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.419393 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.419221    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.419393 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.419314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.419917 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.419893    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.421677 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.421659    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.475156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.475126    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm2t\" (UniqueName: \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\") pod \"maas-api-revoke-sub-28c9d442-tmsxc\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.525062 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.525031    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:12.901174 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.901082    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"]
Oct 30 21:56:12.908566 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:56:12.908534    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21bc0f8_7b0c_43c3_9095_a95cfe7f1c8d.slice/crio-5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08 WatchSource:0}: Error finding container 5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08: Status 404 returned error can't find the container with id 5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08
Oct 30 21:56:12.911274 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.911255    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:56:12.929398 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:12.929362    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc" event={"ID":"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d","Type":"ContainerStarted","Data":"5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08"}
Oct 30 21:56:13.279918 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:56:13.279878    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:56:16.141366 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:16.141337    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:16.225389 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:16.225349    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:56:19.984234 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:19.984198    5510 generic.go:356] "Generic (PLEG): container finished" podID="c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" containerID="3bf6b28f20aed8a2f580f9b719c9090859ee891b64277d9ceb533788dda5a800" exitCode=0
Oct 30 21:56:19.984623 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:19.984290    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc" event={"ID":"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d","Type":"ContainerDied","Data":"3bf6b28f20aed8a2f580f9b719c9090859ee891b64277d9ceb533788dda5a800"}
Oct 30 21:56:20.800838 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:20.800774    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:21.077266 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.077088    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:21.139270 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.139245    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:21.237068 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.237036    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\" (UniqueName: \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\") pod \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") "
Oct 30 21:56:21.237249 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.237084    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\") pod \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") "
Oct 30 21:56:21.237249 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.237115    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\") pod \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\" (UID: \"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d\") "
Oct 30 21:56:21.237544 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.237513    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca" pod "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" (UID: "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:21.237804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.237753    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:21.240047 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.240019    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca" pod "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" (UID: "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:21.240187 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.240168    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t" pod "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" (UID: "c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d"). InnerVolumeSpecName "kube-api-access-zjm2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:21.339288 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.339212    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm2t\" (UniqueName: \"kubernetes.io/projected/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-kube-api-access-zjm2t\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:21.339288 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:21.339238    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.005746 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:22.005710    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc" event={"ID":"c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d","Type":"ContainerDied","Data":"5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08"}
Oct 30 21:56:22.005947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:22.005728    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"
Oct 30 21:56:22.005947 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:22.005801    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5059d73480b7f422ab366801df31aa7be9a543eb699b61944d858b1b06bf8c08"
Oct 30 21:56:22.235117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:22.235072    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:23.527226 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.527195    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 21:56:23.527918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.527896    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" containerName="revoke-keys"
Oct 30 21:56:23.528030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.527918    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" containerName="revoke-keys"
Oct 30 21:56:23.528084 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.528070    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" containerName="revoke-keys"
Oct 30 21:56:23.531703 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.531679    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.560971    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.561028    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8gt\" (UniqueName: \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.561114    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561275 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.561254    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.561291    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.561437 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.561405    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.626010 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.625965    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"premium-w3-f776cb-kserve-self-signed-certs\""
Oct 30 21:56:23.628056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.628027    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 21:56:23.679565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679532    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.679758 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679576    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.679758 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679749    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.680079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679847    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.680079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679897    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.680079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679950    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8gt\" (UniqueName: \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.680079 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.679990    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.681294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.680994    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.683623 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.683601    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.686464 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.686418    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.687986 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.687963    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.728527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.728474    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:23.735550 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.735528    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8gt\" (UniqueName: \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\") pod \"premium-w3-f776cb-kserve-67dcc47b4b-8lzt5\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") " pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:23.844137 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:23.844064    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:24.284047 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:24.284012    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 21:56:24.291119 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:56:24.291095    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b7df0a_69af_4986_8824_3376a157b895.slice/crio-ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195 WatchSource:0}: Error finding container ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195: Status 404 returned error can't find the container with id ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195
Oct 30 21:56:25.022408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:25.022372    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" event={"ID":"99b7df0a-69af-4986-8824-3376a157b895","Type":"ContainerStarted","Data":"8826c335763254851d43b9b50dfd77ec65a08451daae52ff224a63eef4ee8a20"}
Oct 30 21:56:25.022408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:25.022419    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" event={"ID":"99b7df0a-69af-4986-8824-3376a157b895","Type":"ContainerStarted","Data":"ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195"}
Oct 30 21:56:25.141538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:25.141491    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podStartSLOduration=2.141478986 podStartE2EDuration="2.141478986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:25.140435623 +0000 UTC m=+794.124342499" watchObservedRunningTime="2026-10-30 21:56:25.141478986 +0000 UTC m=+794.125385862"
Oct 30 21:56:25.187686 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:25.187653    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:25.844867 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:25.844836    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:29.984420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:29.984384    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:33.845073 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:33.844993    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:33.858257 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:33.858227    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:34.088103 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:34.088070    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 21:56:34.428606 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:34.428574    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:34.578133 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:34.578099    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:38.929460 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:38.929425    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:39.410922 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:39.410891    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:43.280962 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:56:43.280919    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:56:43.444310 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:43.444274    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:48.016906 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:48.016871    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:48.040806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:48.040762    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:56:51.460572 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.460492    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 21:56:51.470155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.470117    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.520251 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.520221    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 21:56:51.535570 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535543    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535590    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535672    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrmh\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535690    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535722    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535757    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535813    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535855    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.535895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.535879    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.563484 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.563456    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f776cb-openshift-default-dockercfg-gj5q6\""
Oct 30 21:56:51.636509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636481    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636534    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636560    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636584    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636613    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636665    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrmh\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636689    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636739    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.636926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636815    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.637131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.636932    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.637131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.637088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.637237 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.637186    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.637292 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.637249    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.637526 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.637509    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.638872 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.638851    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.639113 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.639093    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.690246 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.690213    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.690388 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.690283    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrmh\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:51.804929 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:51.804837    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:52.232092 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:52.232063    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 21:56:52.238883 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:56:52.238859    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfd46b8_29bf_4bb4_b149_936edecc0429.slice/crio-d8455a32cc821a772b5ed6a4a1678329bb4f18923a74ca6844a3bffe67f89a7c WatchSource:0}: Error finding container d8455a32cc821a772b5ed6a4a1678329bb4f18923a74ca6844a3bffe67f89a7c: Status 404 returned error can't find the container with id d8455a32cc821a772b5ed6a4a1678329bb4f18923a74ca6844a3bffe67f89a7c
Oct 30 21:56:52.241158 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:52.241125    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:56:52.241238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:52.241189    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:56:52.241238 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:52.241217    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:56:52.534821 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:52.534671    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:53.198996 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:53.198959    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" event={"ID":"ecfd46b8-29bf-4bb4-b149-936edecc0429","Type":"ContainerStarted","Data":"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"}
Oct 30 21:56:53.199212 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:53.199013    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" event={"ID":"ecfd46b8-29bf-4bb4-b149-936edecc0429","Type":"ContainerStarted","Data":"d8455a32cc821a772b5ed6a4a1678329bb4f18923a74ca6844a3bffe67f89a7c"}
Oct 30 21:56:53.805941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:53.805907    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:53.807744 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:53.807713    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.807902 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:53.807775    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:54.805455 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:54.805420    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.805635 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:54.805475    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:55.805371 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:55.805340    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.805765 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:55.805398    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:56.805328 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:56.805286    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.805536 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:56.805360    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:57.001452 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.001395    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podStartSLOduration=6.001381002 podStartE2EDuration="6.001381002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:53.36982904 +0000 UTC m=+822.353735922" watchObservedRunningTime="2026-10-30 21:56:57.001381002 +0000 UTC m=+825.985287878"
Oct 30 21:56:57.002721 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.002697    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"]
Oct 30 21:56:57.006771 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.006754    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.068745 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.068632    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:57.115478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.114119    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:57.115478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.114234    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:56:57.115478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.114865    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:57.116928 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.116906    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"]
Oct 30 21:56:57.123819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.123770    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.123952 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.123878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.124066 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.124046    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.128965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.128943    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:57.225858 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.225828    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.226277 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.226253    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.226405 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.226389    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.227301 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.227274    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.228949 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.228926    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.281802 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.280577    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-59c879e6-mph9h\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.320968 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.320899    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:56:57.717160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.717131    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"]
Oct 30 21:56:57.724563 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:56:57.724536    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62a2a50_94c4_4973_a64a_bc738e4c3865.slice/crio-36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230 WatchSource:0}: Error finding container 36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230: Status 404 returned error can't find the container with id 36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230
Oct 30 21:56:57.805411 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.805384    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:57.805554 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:57.805429    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:58.229051 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:58.229011    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h" event={"ID":"e62a2a50-94c4-4973-a64a-bc738e4c3865","Type":"ContainerStarted","Data":"7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3"}
Oct 30 21:56:58.229216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:58.229075    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h" event={"ID":"e62a2a50-94c4-4973-a64a-bc738e4c3865","Type":"ContainerStarted","Data":"36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230"}
Oct 30 21:56:58.336307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:58.336197    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h" podStartSLOduration=2.336183769 podStartE2EDuration="2.336183769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:58.333957865 +0000 UTC m=+827.317864740" watchObservedRunningTime="2026-10-30 21:56:58.336183769 +0000 UTC m=+827.320090644"
Oct 30 21:56:58.805446 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:58.805412    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:56:58.805704 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:58.805466    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 21:56:59.078848 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.078744    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:59.239860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.239818    5510 generic.go:356] "Generic (PLEG): container finished" podID="e62a2a50-94c4-4973-a64a-bc738e4c3865" containerID="7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3" exitCode=0
Oct 30 21:56:59.240038 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.239949    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h" event={"ID":"e62a2a50-94c4-4973-a64a-bc738e4c3865","Type":"ContainerDied","Data":"7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3"}
Oct 30 21:56:59.809495 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.809471    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:59.809914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.809870    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:56:59.810437 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:56:59.810417    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 21:57:00.390233 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.390207    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:57:00.465212 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.465180    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\") pod \"e62a2a50-94c4-4973-a64a-bc738e4c3865\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") "
Oct 30 21:57:00.465372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.465226    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\") pod \"e62a2a50-94c4-4973-a64a-bc738e4c3865\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") "
Oct 30 21:57:00.465372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.465288    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\") pod \"e62a2a50-94c4-4973-a64a-bc738e4c3865\" (UID: \"e62a2a50-94c4-4973-a64a-bc738e4c3865\") "
Oct 30 21:57:00.465615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.465591    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca" pod "e62a2a50-94c4-4973-a64a-bc738e4c3865" (UID: "e62a2a50-94c4-4973-a64a-bc738e4c3865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:00.465832 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.465816    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:00.467311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.467288    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd" pod "e62a2a50-94c4-4973-a64a-bc738e4c3865" (UID: "e62a2a50-94c4-4973-a64a-bc738e4c3865"). InnerVolumeSpecName "kube-api-access-xfrtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:00.467373 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.467354    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca" pod "e62a2a50-94c4-4973-a64a-bc738e4c3865" (UID: "e62a2a50-94c4-4973-a64a-bc738e4c3865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:00.566910 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.566824    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/e62a2a50-94c4-4973-a64a-bc738e4c3865-kube-api-access-xfrtd\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:00.566910 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:00.566855    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62a2a50-94c4-4973-a64a-bc738e4c3865-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:01.261377 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:01.261346    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"
Oct 30 21:57:01.261377 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:01.261356    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h" event={"ID":"e62a2a50-94c4-4973-a64a-bc738e4c3865","Type":"ContainerDied","Data":"36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230"}
Oct 30 21:57:01.261830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:01.261426    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c10af977cf1383443544a7be968684b49942eaeec2c5df2108854da0f67230"
Oct 30 21:57:01.535918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:01.535883    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:06.032668 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.032622    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"]
Oct 30 21:57:06.033409 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.033387    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e62a2a50-94c4-4973-a64a-bc738e4c3865" containerName="revoke-keys"
Oct 30 21:57:06.033503 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.033411    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62a2a50-94c4-4973-a64a-bc738e4c3865" containerName="revoke-keys"
Oct 30 21:57:06.033581 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.033568    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62a2a50-94c4-4973-a64a-bc738e4c3865" containerName="revoke-keys"
Oct 30 21:57:06.036980 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.036959    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.085432 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.084334    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:06.089813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.088961    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:06.129242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.129061    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985hv\" (UniqueName: \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.129242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.129170    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.129242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.129206    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.135134 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.135111    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:06.135286 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.135125    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:06.135286 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.135177    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:06.137353 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.137331    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"]
Oct 30 21:57:06.230735 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.230697    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.230931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.230747    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.231014 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.230996    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985hv\" (UniqueName: \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.232049 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.232019    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.234393 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.234362    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.321522 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.321441    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-985hv\" (UniqueName: \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\") pod \"maas-api-revoke-sub-49606ce5-22sp6\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.350146 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.350107    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:06.789458 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:06.789416    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"]
Oct 30 21:57:06.797521 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:06.797486    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9fb28c_9e7c_4909_a0c5_bdc61c4153b5.slice/crio-498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0 WatchSource:0}: Error finding container 498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0: Status 404 returned error can't find the container with id 498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0
Oct 30 21:57:07.322017 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:07.321910    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6" event={"ID":"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5","Type":"ContainerStarted","Data":"68ba9afefb42cab8f11adf4bbd99355e22d3d9a8e827ab0df16aabe5d21cc7f0"}
Oct 30 21:57:07.322017 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:07.321970    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6" event={"ID":"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5","Type":"ContainerStarted","Data":"498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0"}
Oct 30 21:57:07.431524 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:07.431465    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6" podStartSLOduration=2.431451283 podStartE2EDuration="2.431451283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:07.429860552 +0000 UTC m=+836.413767429" watchObservedRunningTime="2026-10-30 21:57:07.431451283 +0000 UTC m=+836.415358200"
Oct 30 21:57:08.329123 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:08.329091    5510 generic.go:356] "Generic (PLEG): container finished" podID="0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" containerID="68ba9afefb42cab8f11adf4bbd99355e22d3d9a8e827ab0df16aabe5d21cc7f0" exitCode=0
Oct 30 21:57:08.329521 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:08.329175    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6" event={"ID":"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5","Type":"ContainerDied","Data":"68ba9afefb42cab8f11adf4bbd99355e22d3d9a8e827ab0df16aabe5d21cc7f0"}
Oct 30 21:57:09.477056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.477029    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:09.570691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.570660    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\" (UniqueName: \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\") pod \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") "
Oct 30 21:57:09.570897 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.570803    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\") pod \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") "
Oct 30 21:57:09.570897 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.570872    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\") pod \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\" (UID: \"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5\") "
Oct 30 21:57:09.571161 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.571127    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca" pod "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" (UID: "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:09.572838 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.572815    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca" pod "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" (UID: "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:09.572937 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.572837    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv" pod "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" (UID: "0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5"). InnerVolumeSpecName "kube-api-access-985hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:09.672689 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.672614    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:09.672689 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.672644    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:09.672689 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:09.672654    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-985hv\" (UniqueName: \"kubernetes.io/projected/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5-kube-api-access-985hv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:10.342399 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:10.342369    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"
Oct 30 21:57:10.342564 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:10.342397    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6" event={"ID":"0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5","Type":"ContainerDied","Data":"498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0"}
Oct 30 21:57:10.342564 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:10.342447    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="498aa2536a44f891784a641a276eb7d9d4c65a5072b3d5eaffabeee683338ef0"
Oct 30 21:57:10.765661 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:10.765626    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:13.279710 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:57:13.279667    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:57:15.203800 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:15.203744    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:17.135098 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.135069    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"]
Oct 30 21:57:17.135572 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.135558    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" containerName="revoke-keys"
Oct 30 21:57:17.135615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.135572    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" containerName="revoke-keys"
Oct 30 21:57:17.135678 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.135668    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" containerName="revoke-keys"
Oct 30 21:57:17.139240 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.139222    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.188627 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.188596    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:17.234465 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.234436    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:17.234619 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.234486    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:17.234619 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.234539    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:17.236852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.236821    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"]
Oct 30 21:57:17.254729 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.254698    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.254928 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.254910    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2gz\" (UniqueName: \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.255002 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.254967    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.355961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.355929    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.356113 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.356035    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2gz\" (UniqueName: \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.356113 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.356072    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.356731 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.356697    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.358282 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.358255    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.409815 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.409708    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2gz\" (UniqueName: \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\") pod \"maas-api-revoke-sub-21f18ef5-qwj7v\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.452201 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.452170    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:17.824407 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:17.824373    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"]
Oct 30 21:57:17.830872 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:17.830847    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2317202d_e739_4622_9aff_3b36b6f6d501.slice/crio-17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc WatchSource:0}: Error finding container 17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc: Status 404 returned error can't find the container with id 17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc
Oct 30 21:57:18.399280 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:18.399247    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v" event={"ID":"2317202d-e739-4622-9aff-3b36b6f6d501","Type":"ContainerStarted","Data":"0bc28b50001d450f12214c53c8a1dec00a9a9d8251c66bc7c6adc487bf02890b"}
Oct 30 21:57:18.399733 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:18.399309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v" event={"ID":"2317202d-e739-4622-9aff-3b36b6f6d501","Type":"ContainerStarted","Data":"17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc"}
Oct 30 21:57:18.510623 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:18.510572    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v" podStartSLOduration=1.510558584 podStartE2EDuration="1.510558584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:18.508563265 +0000 UTC m=+847.492470140" watchObservedRunningTime="2026-10-30 21:57:18.510558584 +0000 UTC m=+847.494465460"
Oct 30 21:57:19.407115 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:19.407076    5510 generic.go:356] "Generic (PLEG): container finished" podID="2317202d-e739-4622-9aff-3b36b6f6d501" containerID="0bc28b50001d450f12214c53c8a1dec00a9a9d8251c66bc7c6adc487bf02890b" exitCode=0
Oct 30 21:57:19.407548 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:19.407127    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v" event={"ID":"2317202d-e739-4622-9aff-3b36b6f6d501","Type":"ContainerDied","Data":"0bc28b50001d450f12214c53c8a1dec00a9a9d8251c66bc7c6adc487bf02890b"}
Oct 30 21:57:19.615594 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:19.615563    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:20.558924 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.558895    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:20.700963 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.700877    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\") pod \"2317202d-e739-4622-9aff-3b36b6f6d501\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") "
Oct 30 21:57:20.701130 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.700967    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\") pod \"2317202d-e739-4622-9aff-3b36b6f6d501\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") "
Oct 30 21:57:20.701130 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.701063    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\" (UniqueName: \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\") pod \"2317202d-e739-4622-9aff-3b36b6f6d501\" (UID: \"2317202d-e739-4622-9aff-3b36b6f6d501\") "
Oct 30 21:57:20.701261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.701229    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca" pod "2317202d-e739-4622-9aff-3b36b6f6d501" (UID: "2317202d-e739-4622-9aff-3b36b6f6d501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:20.701375 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.701359    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:20.703931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.703900    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz" pod "2317202d-e739-4622-9aff-3b36b6f6d501" (UID: "2317202d-e739-4622-9aff-3b36b6f6d501"). InnerVolumeSpecName "kube-api-access-tr2gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:20.706926 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.706899    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca" pod "2317202d-e739-4622-9aff-3b36b6f6d501" (UID: "2317202d-e739-4622-9aff-3b36b6f6d501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:20.802249 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.802214    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2317202d-e739-4622-9aff-3b36b6f6d501-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:20.802249 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:20.802245    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr2gz\" (UniqueName: \"kubernetes.io/projected/2317202d-e739-4622-9aff-3b36b6f6d501-kube-api-access-tr2gz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:21.312139 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.312110    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:21.419704 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.419631    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"
Oct 30 21:57:21.419704 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.419657    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v" event={"ID":"2317202d-e739-4622-9aff-3b36b6f6d501","Type":"ContainerDied","Data":"17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc"}
Oct 30 21:57:21.419704 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.419703    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c81ce7f2307f938d1f595c6c431908693810cc942892a7cd416fdfd3a7fabc"
Oct 30 21:57:21.430647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.430624    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"]
Oct 30 21:57:21.431119 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.431105    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2317202d-e739-4622-9aff-3b36b6f6d501" containerName="revoke-keys"
Oct 30 21:57:21.431175 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.431120    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2317202d-e739-4622-9aff-3b36b6f6d501" containerName="revoke-keys"
Oct 30 21:57:21.431235 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.431225    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2317202d-e739-4622-9aff-3b36b6f6d501" containerName="revoke-keys"
Oct 30 21:57:21.434279 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.434251    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.484941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.484894    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:21.484941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.484928    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:21.485155 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.484995    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:21.510599 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.510568    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.510798 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.510611    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252fd\" (UniqueName: \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.510865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.510818    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.517217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.517188    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:21.520989 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.520957    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"]
Oct 30 21:57:21.611967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.611934    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252fd\" (UniqueName: \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.612433 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.612018    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.612433 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.612104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.612649 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.612626    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.614387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.614365    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.666139 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.666106    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252fd\" (UniqueName: \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\") pod \"maas-api-revoke-sub-65f425ba-8tg67\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:21.748243 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:21.748217    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:22.124341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:22.124260    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"]
Oct 30 21:57:22.131805 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:22.131748    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be5295a_7bf4_41bc_8f4c_067c8c4f2d31.slice/crio-ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9 WatchSource:0}: Error finding container ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9: Status 404 returned error can't find the container with id ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9
Oct 30 21:57:22.427261 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:22.427183    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67" event={"ID":"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31","Type":"ContainerStarted","Data":"ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9"}
Oct 30 21:57:23.436772 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:23.436691    5510 generic.go:356] "Generic (PLEG): container finished" podID="2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" containerID="de9da4cd1f08b5b2786f96427b9be1d0a6a938d3a029b55239aceeb8cbadd05b" exitCode=0
Oct 30 21:57:23.437181 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:23.436805    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67" event={"ID":"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31","Type":"ContainerDied","Data":"de9da4cd1f08b5b2786f96427b9be1d0a6a938d3a029b55239aceeb8cbadd05b"}
Oct 30 21:57:24.118509 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.118475    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:24.580735 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.580710    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:24.748001 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.747970    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\" (UniqueName: \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\") pod \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") "
Oct 30 21:57:24.748177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.748024    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\") pod \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") "
Oct 30 21:57:24.748177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.748084    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\") pod \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\" (UID: \"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31\") "
Oct 30 21:57:24.748478 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.748455    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca" pod "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" (UID: "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:24.750100 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.750077    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca" pod "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" (UID: "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:24.750199 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.750095    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd" pod "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" (UID: "2be5295a-7bf4-41bc-8f4c-067c8c4f2d31"). InnerVolumeSpecName "kube-api-access-252fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:24.849156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.849122    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252fd\" (UniqueName: \"kubernetes.io/projected/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-kube-api-access-252fd\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:24.849156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.849148    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:24.849319 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:24.849162    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:25.449961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:25.449873    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"
Oct 30 21:57:25.449961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:25.449908    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67" event={"ID":"2be5295a-7bf4-41bc-8f4c-067c8c4f2d31","Type":"ContainerDied","Data":"ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9"}
Oct 30 21:57:25.450156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:25.449974    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca659d6fc6a628a48e4beea806b5d1d2d2f4af7dd151bd1945bceecc114fd9d9"
Oct 30 21:57:27.845391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.845356    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"]
Oct 30 21:57:27.845901 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.845887    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" containerName="revoke-keys"
Oct 30 21:57:27.845944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.845902    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" containerName="revoke-keys"
Oct 30 21:57:27.846007 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.845997    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" containerName="revoke-keys"
Oct 30 21:57:27.849324 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.849299    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:27.944490 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.944458    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:27.944655 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.944509    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:27.944655 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.944572    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:27.944744 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.944672    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:27.947102 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.947076    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"]
Oct 30 21:57:27.984606 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.984574    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:27.984774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.984610    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rng7g\" (UniqueName: \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:27.984774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:27.984707    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.085649 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.085615    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.085862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.085751    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.085862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.085800    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rng7g\" (UniqueName: \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.086554 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.086526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.088511 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.088477    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.138735 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.138668    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rng7g\" (UniqueName: \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\") pod \"maas-api-revoke-sub-50380152-6qftg\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.166672 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.166640    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:28.428171 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.428105    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:57:28.530671 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.530641    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:28.541001 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:28.540971    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"]
Oct 30 21:57:28.548269 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:28.548240    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03912648_c62f_48c3_a2a3_714393eb44e4.slice/crio-f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2 WatchSource:0}: Error finding container f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2: Status 404 returned error can't find the container with id f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2
Oct 30 21:57:29.485826 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:29.485572    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg" event={"ID":"03912648-c62f-48c3-a2a3-714393eb44e4","Type":"ContainerStarted","Data":"1b5ed8649bfe050f3974fd8980ffca9e666b15279fe2cc24a781578e88ede259"}
Oct 30 21:57:29.485826 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:29.485643    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg" event={"ID":"03912648-c62f-48c3-a2a3-714393eb44e4","Type":"ContainerStarted","Data":"f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2"}
Oct 30 21:57:29.598811 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:29.596732    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg" podStartSLOduration=2.59671302 podStartE2EDuration="2.59671302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:29.596190348 +0000 UTC m=+858.580097225" watchObservedRunningTime="2026-10-30 21:57:29.59671302 +0000 UTC m=+858.580619897"
Oct 30 21:57:30.492894 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:30.492858    5510 generic.go:356] "Generic (PLEG): container finished" podID="03912648-c62f-48c3-a2a3-714393eb44e4" containerID="1b5ed8649bfe050f3974fd8980ffca9e666b15279fe2cc24a781578e88ede259" exitCode=0
Oct 30 21:57:30.493277 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:30.492939    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg" event={"ID":"03912648-c62f-48c3-a2a3-714393eb44e4","Type":"ContainerDied","Data":"1b5ed8649bfe050f3974fd8980ffca9e666b15279fe2cc24a781578e88ede259"}
Oct 30 21:57:31.640513 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.640483    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:31.664898 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.664871    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\") pod \"03912648-c62f-48c3-a2a3-714393eb44e4\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") "
Oct 30 21:57:31.665020 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.664959    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\") pod \"03912648-c62f-48c3-a2a3-714393eb44e4\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") "
Oct 30 21:57:31.665060 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.665037    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\" (UniqueName: \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\") pod \"03912648-c62f-48c3-a2a3-714393eb44e4\" (UID: \"03912648-c62f-48c3-a2a3-714393eb44e4\") "
Oct 30 21:57:31.665349 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.665323    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca" pod "03912648-c62f-48c3-a2a3-714393eb44e4" (UID: "03912648-c62f-48c3-a2a3-714393eb44e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:31.666833 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.666808    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca" pod "03912648-c62f-48c3-a2a3-714393eb44e4" (UID: "03912648-c62f-48c3-a2a3-714393eb44e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:31.666979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.666955    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g" pod "03912648-c62f-48c3-a2a3-714393eb44e4" (UID: "03912648-c62f-48c3-a2a3-714393eb44e4"). InnerVolumeSpecName "kube-api-access-rng7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:31.766565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.766496    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.766565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.766522    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03912648-c62f-48c3-a2a3-714393eb44e4-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.766565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:31.766533    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rng7g\" (UniqueName: \"kubernetes.io/projected/03912648-c62f-48c3-a2a3-714393eb44e4-kube-api-access-rng7g\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:32.506848 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:32.506817    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"
Oct 30 21:57:32.506848 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:32.506826    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg" event={"ID":"03912648-c62f-48c3-a2a3-714393eb44e4","Type":"ContainerDied","Data":"f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2"}
Oct 30 21:57:32.507070 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:32.506882    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f908d17224ec06b00312fd1e2e6ec97cb6e9c7636ee0224abd01fe0de7cf4eb2"
Oct 30 21:57:33.055806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:33.054600    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:37.231723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.231691    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 21:57:37.232258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.232241    5510 cpu_manager.go:414] "containerMap: removing container" podUID="03912648-c62f-48c3-a2a3-714393eb44e4" containerName="revoke-keys"
Oct 30 21:57:37.232303 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.232260    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03912648-c62f-48c3-a2a3-714393eb44e4" containerName="revoke-keys"
Oct 30 21:57:37.232363 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.232353    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="03912648-c62f-48c3-a2a3-714393eb44e4" containerName="revoke-keys"
Oct 30 21:57:37.235357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.235340    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.283939 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.283915    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-17d809-openshift-default-dockercfg-ppc95\""
Oct 30 21:57:37.327751 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.327719    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.327800    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.327929    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.327984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn9j\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328287 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.328160    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328287 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.328201    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.328290    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.328348    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.328391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.328381    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.333552 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.333529    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 21:57:37.429491 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429456    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429491 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429495    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429525    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429565    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429589    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429656    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429713 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429682    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429969 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429717    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.429969 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.429756    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn9j\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.430083 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.430014    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.430185 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.430163    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.430251 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.430191    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.430302 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.430253    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.430468 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.430445    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.431929 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.431901    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.432215 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.432196    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.486048 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.485971    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.486179 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.486075    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn9j\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:37.524077 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.524032    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:37.570527 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:37.548392    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:38.002110 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.002070    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 21:57:38.008332 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:38.008307    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2793053a_78e8_43d1_87e3_9f753ecd6e89.slice/crio-c26f04cd53e4781a2e0f64adb8c0b83f955edcde08b43b17283575a7b91629b2 WatchSource:0}: Error finding container c26f04cd53e4781a2e0f64adb8c0b83f955edcde08b43b17283575a7b91629b2: Status 404 returned error can't find the container with id c26f04cd53e4781a2e0f64adb8c0b83f955edcde08b43b17283575a7b91629b2
Oct 30 21:57:38.010540 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.010513    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:57:38.010607 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.010576    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:57:38.010642 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.010606    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:57:38.543552 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.543519    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" event={"ID":"2793053a-78e8-43d1-87e3-9f753ecd6e89","Type":"ContainerStarted","Data":"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"}
Oct 30 21:57:38.543987 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.543569    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" event={"ID":"2793053a-78e8-43d1-87e3-9f753ecd6e89","Type":"ContainerStarted","Data":"c26f04cd53e4781a2e0f64adb8c0b83f955edcde08b43b17283575a7b91629b2"}
Oct 30 21:57:38.548961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:38.548944    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:39.174804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:39.169407    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:39.326667 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:39.326631    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:39.550566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:39.550530    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 21:57:39.551029 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:39.550589    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 21:57:40.549180 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:40.549151    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 21:57:40.549352 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:40.549196    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 21:57:41.550031 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:41.549984    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 21:57:41.550481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:41.550055    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 21:57:42.006542 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:42.006467    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:42.549307 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:42.549274    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 21:57:42.549484 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:42.549331    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 21:57:43.309802 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:57:43.309256    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:57:43.549819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:43.549766    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 21:57:43.549974 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:43.549838    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 21:57:44.618398 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:44.618362    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:44.618894 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:44.618687    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:44.619500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:44.619483    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 21:57:44.777237 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:44.777180    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podStartSLOduration=7.777166848 podStartE2EDuration="7.777166848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:38.680115108 +0000 UTC m=+867.664021995" watchObservedRunningTime="2026-10-30 21:57:44.777166848 +0000 UTC m=+873.761073724"
Oct 30 21:57:46.534032 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:46.533987    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:50.723021 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.722987    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"]
Oct 30 21:57:50.727106 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.727085    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.779335 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.779307    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:50.779335 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.779329    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:50.779535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.779354    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:50.812485 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.812420    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.812485 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.812479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.812684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.812565    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjqb\" (UniqueName: \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.825203 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.825182    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:57:50.827724 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.827699    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"]
Oct 30 21:57:50.841619 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.841594    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"]
Oct 30 21:57:50.846616 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.846595    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:50.898314 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.898281    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"]
Oct 30 21:57:50.913952 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.913911    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.914104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.913971    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.914104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.914093    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjqb\" (UniqueName: \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.914547 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.914526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.916350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.916326    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:50.966825 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:50.966798    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjqb\" (UniqueName: \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\") pod \"maas-api-revoke-sub-4f40b463-rpcmp\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:51.015076 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.014994    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.015076 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.015046    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.015247 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.015140    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdzm\" (UniqueName: \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.032618 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.032592    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:51.040537 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.040518    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:51.118641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.117412    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.118641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.117461    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.118641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.117508    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdzm\" (UniqueName: \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.118641 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.118222    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.120866 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.120841    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.175538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.175464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdzm\" (UniqueName: \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\") pod \"maas-api-revoke-sub-02a51a84-qd8d9\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.461221 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.461188    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:51.476152 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.476119    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"]
Oct 30 21:57:51.489700 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:51.483239    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b14ec5_74f6_4ae3_b4a5_3c45ffb835fc.slice/crio-11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b WatchSource:0}: Error finding container 11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b: Status 404 returned error can't find the container with id 11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b
Oct 30 21:57:51.653025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.652991    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp" event={"ID":"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc","Type":"ContainerStarted","Data":"11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b"}
Oct 30 21:57:51.842695 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:51.842661    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"]
Oct 30 21:57:51.849285 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:57:51.849261    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1054b5_ab67_47ff_99bb_84382e3f1492.slice/crio-a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5 WatchSource:0}: Error finding container a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5: Status 404 returned error can't find the container with id a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5
Oct 30 21:57:52.669262 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:52.669219    5510 generic.go:356] "Generic (PLEG): container finished" podID="35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" containerID="83e6e0cd2bf60693483d7e0bd5fc8b8c757a7f2b2c5ec89d97934f640d1fe1b3" exitCode=0
Oct 30 21:57:52.669431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:52.669297    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp" event={"ID":"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc","Type":"ContainerDied","Data":"83e6e0cd2bf60693483d7e0bd5fc8b8c757a7f2b2c5ec89d97934f640d1fe1b3"}
Oct 30 21:57:52.670862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:52.670839    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9" event={"ID":"3e1054b5-ab67-47ff-99bb-84382e3f1492","Type":"ContainerStarted","Data":"02dcbc31c4406a08be9e8269a6562f36fea7a78c55f9a921cbdb90e8b4243d47"}
Oct 30 21:57:52.670971 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:52.670881    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9" event={"ID":"3e1054b5-ab67-47ff-99bb-84382e3f1492","Type":"ContainerStarted","Data":"a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5"}
Oct 30 21:57:52.881487 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:52.881390    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9" podStartSLOduration=2.881375147 podStartE2EDuration="2.881375147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:52.87956538 +0000 UTC m=+881.863472256" watchObservedRunningTime="2026-10-30 21:57:52.881375147 +0000 UTC m=+881.865282022"
Oct 30 21:57:53.678391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.678358    5510 generic.go:356] "Generic (PLEG): container finished" podID="3e1054b5-ab67-47ff-99bb-84382e3f1492" containerID="02dcbc31c4406a08be9e8269a6562f36fea7a78c55f9a921cbdb90e8b4243d47" exitCode=0
Oct 30 21:57:53.678575 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.678448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9" event={"ID":"3e1054b5-ab67-47ff-99bb-84382e3f1492","Type":"ContainerDied","Data":"02dcbc31c4406a08be9e8269a6562f36fea7a78c55f9a921cbdb90e8b4243d47"}
Oct 30 21:57:53.837479 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.837443    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:53.949114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.947982    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\") pod \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") "
Oct 30 21:57:53.949114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.948109    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\" (UniqueName: \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\") pod \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") "
Oct 30 21:57:53.949114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.948193    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\") pod \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\" (UID: \"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc\") "
Oct 30 21:57:53.949114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.948947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca" pod "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" (UID: "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:53.954815 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.953899    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca" pod "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" (UID: "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:53.954815 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:53.954304    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb" pod "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" (UID: "35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc"). InnerVolumeSpecName "kube-api-access-bxjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:54.050114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.050080    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.050114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.050109    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.050114 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.050120    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxjqb\" (UniqueName: \"kubernetes.io/projected/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc-kube-api-access-bxjqb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.685681 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.685649    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"
Oct 30 21:57:54.685681 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.685677    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp" event={"ID":"35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc","Type":"ContainerDied","Data":"11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b"}
Oct 30 21:57:54.685918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.685723    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11afa9a3740eae4e84d214afa949d9ee54a9cae04a73fe425c60aec5a6b8db0b"
Oct 30 21:57:54.833294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.833270    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:54.858846 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.858816    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\") pod \"3e1054b5-ab67-47ff-99bb-84382e3f1492\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") "
Oct 30 21:57:54.858967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.858856    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\" (UniqueName: \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\") pod \"3e1054b5-ab67-47ff-99bb-84382e3f1492\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") "
Oct 30 21:57:54.858967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.858958    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\") pod \"3e1054b5-ab67-47ff-99bb-84382e3f1492\" (UID: \"3e1054b5-ab67-47ff-99bb-84382e3f1492\") "
Oct 30 21:57:54.859198 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.859148    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca" pod "3e1054b5-ab67-47ff-99bb-84382e3f1492" (UID: "3e1054b5-ab67-47ff-99bb-84382e3f1492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:54.859470 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.859437    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.861036 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.861012    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm" pod "3e1054b5-ab67-47ff-99bb-84382e3f1492" (UID: "3e1054b5-ab67-47ff-99bb-84382e3f1492"). InnerVolumeSpecName "kube-api-access-9gdzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:54.861204 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.861179    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca" pod "3e1054b5-ab67-47ff-99bb-84382e3f1492" (UID: "3e1054b5-ab67-47ff-99bb-84382e3f1492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:54.960797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.960757    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdzm\" (UniqueName: \"kubernetes.io/projected/3e1054b5-ab67-47ff-99bb-84382e3f1492-kube-api-access-9gdzm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:54.960797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:54.960798    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e1054b5-ab67-47ff-99bb-84382e3f1492-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:55.434244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:55.434153    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:55.692644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:55.692561    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9" event={"ID":"3e1054b5-ab67-47ff-99bb-84382e3f1492","Type":"ContainerDied","Data":"a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5"}
Oct 30 21:57:55.692644 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:55.692644    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4255b5501f36e9ba1ad2f50994a63361865a6c4fa71974ad23cac3bdd8431c5"
Oct 30 21:57:55.692863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:55.692580    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"
Oct 30 21:57:59.928141 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:57:59.928108    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:02.429319 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.429287    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"]
Oct 30 21:58:02.430029 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430006    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3e1054b5-ab67-47ff-99bb-84382e3f1492" containerName="revoke-keys"
Oct 30 21:58:02.430138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430029    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1054b5-ab67-47ff-99bb-84382e3f1492" containerName="revoke-keys"
Oct 30 21:58:02.430138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430055    5510 cpu_manager.go:414] "containerMap: removing container" podUID="35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" containerName="revoke-keys"
Oct 30 21:58:02.430138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430062    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" containerName="revoke-keys"
Oct 30 21:58:02.430294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430252    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1054b5-ab67-47ff-99bb-84382e3f1492" containerName="revoke-keys"
Oct 30 21:58:02.430294 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.430268    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" containerName="revoke-keys"
Oct 30 21:58:02.433738 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.433718    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.465158 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.465115    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vx2\" (UniqueName: \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.465326 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.465301    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.465420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.465404    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.482536 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.482460    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:02.485923 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.485896    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"]
Oct 30 21:58:02.528750 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.528718    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:02.528750 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.528721    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:02.528750 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.528748    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:02.566878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.566840    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vx2\" (UniqueName: \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.567075 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.566957    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.567075 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.567004    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.567708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.567681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.569381 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.569356    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.620306 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.620275    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vx2\" (UniqueName: \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\") pod \"maas-api-revoke-sub-98287cab-nlg9m\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:02.747204 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:02.747171    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:03.125375 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:03.125283    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"]
Oct 30 21:58:03.129686 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:03.129653    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439eff63_80e5_4590_b5b4_049ba69e2ce0.slice/crio-bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe WatchSource:0}: Error finding container bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe: Status 404 returned error can't find the container with id bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe
Oct 30 21:58:03.743024 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:03.742943    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m" event={"ID":"439eff63-80e5-4590-b5b4-049ba69e2ce0","Type":"ContainerStarted","Data":"5988b10784ae0d33300356745cbeab7aa981e63dce84fd5243785d96437001df"}
Oct 30 21:58:03.743024 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:03.743000    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m" event={"ID":"439eff63-80e5-4590-b5b4-049ba69e2ce0","Type":"ContainerStarted","Data":"bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe"}
Oct 30 21:58:03.863836 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:03.863763    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m" podStartSLOduration=1.863750281 podStartE2EDuration="1.863750281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:03.861689112 +0000 UTC m=+892.845595988" watchObservedRunningTime="2026-10-30 21:58:03.863750281 +0000 UTC m=+892.847657156"
Oct 30 21:58:04.285018 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:04.284920    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:04.422693 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:04.422661    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:58:04.768417 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:04.768382    5510 generic.go:356] "Generic (PLEG): container finished" podID="439eff63-80e5-4590-b5b4-049ba69e2ce0" containerID="5988b10784ae0d33300356745cbeab7aa981e63dce84fd5243785d96437001df" exitCode=0
Oct 30 21:58:04.769312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:04.769277    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m" event={"ID":"439eff63-80e5-4590-b5b4-049ba69e2ce0","Type":"ContainerDied","Data":"5988b10784ae0d33300356745cbeab7aa981e63dce84fd5243785d96437001df"}
Oct 30 21:58:06.577019 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.576942    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:06.601117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.600124    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\") pod \"439eff63-80e5-4590-b5b4-049ba69e2ce0\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") "
Oct 30 21:58:06.601117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.600274    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\") pod \"439eff63-80e5-4590-b5b4-049ba69e2ce0\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") "
Oct 30 21:58:06.601117 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.600350    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\" (UniqueName: \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\") pod \"439eff63-80e5-4590-b5b4-049ba69e2ce0\" (UID: \"439eff63-80e5-4590-b5b4-049ba69e2ce0\") "
Oct 30 21:58:06.605007 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.604978    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca" pod "439eff63-80e5-4590-b5b4-049ba69e2ce0" (UID: "439eff63-80e5-4590-b5b4-049ba69e2ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:06.620991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.620932    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2" pod "439eff63-80e5-4590-b5b4-049ba69e2ce0" (UID: "439eff63-80e5-4590-b5b4-049ba69e2ce0"). InnerVolumeSpecName "kube-api-access-r5vx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:06.660805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.658533    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca" pod "439eff63-80e5-4590-b5b4-049ba69e2ce0" (UID: "439eff63-80e5-4590-b5b4-049ba69e2ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:06.704674 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.704597    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:06.704674 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.704630    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439eff63-80e5-4590-b5b4-049ba69e2ce0-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:06.704674 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.704646    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vx2\" (UniqueName: \"kubernetes.io/projected/439eff63-80e5-4590-b5b4-049ba69e2ce0-kube-api-access-r5vx2\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:06.945874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.932922    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m" event={"ID":"439eff63-80e5-4590-b5b4-049ba69e2ce0","Type":"ContainerDied","Data":"bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe"}
Oct 30 21:58:06.945874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.932989    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc71e3ef32fc35f3ae8bc4cc30a58f40dbfeae0812c5b8c6bf798ed8e7c7c6fe"
Oct 30 21:58:06.945874 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:06.933068    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"
Oct 30 21:58:08.488803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.474456    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"]
Oct 30 21:58:08.490274 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.490244    5510 cpu_manager.go:414] "containerMap: removing container" podUID="439eff63-80e5-4590-b5b4-049ba69e2ce0" containerName="revoke-keys"
Oct 30 21:58:08.490441 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.490425    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439eff63-80e5-4590-b5b4-049ba69e2ce0" containerName="revoke-keys"
Oct 30 21:58:08.492620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.492600    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="439eff63-80e5-4590-b5b4-049ba69e2ce0" containerName="revoke-keys"
Oct 30 21:58:08.515877 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.511411    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.549223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.549194    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"]
Oct 30 21:58:08.616755 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.615639    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:08.616755 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.616342    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.616755 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.616477    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.616755 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.616538    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9zg\" (UniqueName: \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.632893 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.632840    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:08.633311 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.633291    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:08.633620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.633604    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:08.717681 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.717637    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9zg\" (UniqueName: \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.717868 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.717809    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.718061 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.718040    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.718537 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.718507    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.720700 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.720677    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.771543 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.771484    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9zg\" (UniqueName: \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\") pod \"maas-api-revoke-sub-753513c8-6vc4j\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:08.788715 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.788690    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:08.840039 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:08.840007    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:09.226108 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:09.226061    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"]
Oct 30 21:58:09.233667 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:09.233638    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27893681_7aff_4f5c_9fec_f062866698be.slice/crio-2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105 WatchSource:0}: Error finding container 2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105: Status 404 returned error can't find the container with id 2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105
Oct 30 21:58:09.956554 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:09.956458    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j" event={"ID":"27893681-7aff-4f5c-9fec-f062866698be","Type":"ContainerStarted","Data":"ed4ff87abad1afc10c5e577cf54919e429dafad95a7d483792a94a2b4539be6a"}
Oct 30 21:58:09.956554 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:09.956525    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j" event={"ID":"27893681-7aff-4f5c-9fec-f062866698be","Type":"ContainerStarted","Data":"2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105"}
Oct 30 21:58:10.063224 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:10.063171    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j" podStartSLOduration=2.063158278 podStartE2EDuration="2.063158278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:10.061966969 +0000 UTC m=+899.045873835" watchObservedRunningTime="2026-10-30 21:58:10.063158278 +0000 UTC m=+899.047065170"
Oct 30 21:58:10.963991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:10.963959    5510 generic.go:356] "Generic (PLEG): container finished" podID="27893681-7aff-4f5c-9fec-f062866698be" containerID="ed4ff87abad1afc10c5e577cf54919e429dafad95a7d483792a94a2b4539be6a" exitCode=0
Oct 30 21:58:10.964391 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:10.963994    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j" event={"ID":"27893681-7aff-4f5c-9fec-f062866698be","Type":"ContainerDied","Data":"ed4ff87abad1afc10c5e577cf54919e429dafad95a7d483792a94a2b4539be6a"}
Oct 30 21:58:11.260689 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:11.260610    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:58:11.262349 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:11.262320    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 21:58:12.124868 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.124829    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:12.263891 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.263806    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\") pod \"27893681-7aff-4f5c-9fec-f062866698be\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") "
Oct 30 21:58:12.264054 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.263920    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\" (UniqueName: \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\") pod \"27893681-7aff-4f5c-9fec-f062866698be\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") "
Oct 30 21:58:12.264054 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.263974    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\") pod \"27893681-7aff-4f5c-9fec-f062866698be\" (UID: \"27893681-7aff-4f5c-9fec-f062866698be\") "
Oct 30 21:58:12.265810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.265763    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca" pod "27893681-7aff-4f5c-9fec-f062866698be" (UID: "27893681-7aff-4f5c-9fec-f062866698be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:12.266129 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.266101    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg" pod "27893681-7aff-4f5c-9fec-f062866698be" (UID: "27893681-7aff-4f5c-9fec-f062866698be"). InnerVolumeSpecName "kube-api-access-cf9zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:12.266520 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.266500    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca" pod "27893681-7aff-4f5c-9fec-f062866698be" (UID: "27893681-7aff-4f5c-9fec-f062866698be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:12.365304 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.365265    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27893681-7aff-4f5c-9fec-f062866698be-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:12.365304 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.365299    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9zg\" (UniqueName: \"kubernetes.io/projected/27893681-7aff-4f5c-9fec-f062866698be-kube-api-access-cf9zg\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:12.365485 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.365313    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27893681-7aff-4f5c-9fec-f062866698be-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:12.893056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.893023    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:12.977702 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.977661    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j" event={"ID":"27893681-7aff-4f5c-9fec-f062866698be","Type":"ContainerDied","Data":"2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105"}
Oct 30 21:58:12.977702 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.977687    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"
Oct 30 21:58:12.977920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:12.977712    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2945d91697f3f521454d298e37ea5b66edf0d774b02545e4dcf29659cb9cd105"
Oct 30 21:58:13.227664 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:13.227633    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:13.307598 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:58:13.307564    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:58:16.270669 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.270635    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"]
Oct 30 21:58:16.271178 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.271163    5510 cpu_manager.go:414] "containerMap: removing container" podUID="27893681-7aff-4f5c-9fec-f062866698be" containerName="revoke-keys"
Oct 30 21:58:16.271229 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.271179    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27893681-7aff-4f5c-9fec-f062866698be" containerName="revoke-keys"
Oct 30 21:58:16.271288 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.271278    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="27893681-7aff-4f5c-9fec-f062866698be" containerName="revoke-keys"
Oct 30 21:58:16.274715 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.274696    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.312319 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.312289    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mnk\" (UniqueName: \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.312461 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.312332    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.312461 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.312423    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.328865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.328838    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"]
Oct 30 21:58:16.370396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.370327    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:16.370396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.370326    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:16.370396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.370362    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:16.370396 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.370337    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:16.413918 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.413884    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mnk\" (UniqueName: \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.414072 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.413931    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.414072 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.413973    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.414723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.414703    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.416315 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.416290    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.467744 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.467712    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mnk\" (UniqueName: \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\") pod \"maas-api-revoke-sub-30ec762a-xgtws\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.587650 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.587568    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:16.639677 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.639646    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"]
Oct 30 21:58:16.644354 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.644333    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.702466 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.702405    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"]
Oct 30 21:58:16.717364 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.717336    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.717497 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.717387    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.717576 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.717555    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgzc\" (UniqueName: \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.818480 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.818448    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.818650 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.818504    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.818650 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.818598    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgzc\" (UniqueName: \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.819124 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.819096    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.820977 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.820948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.872091 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.872030    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgzc\" (UniqueName: \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\") pod \"maas-api-revoke-sub-cad7745c-5dj24\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.958999 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.958961    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"]
Oct 30 21:58:16.964062 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:16.964040    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:16.964178 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:16.964106    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode995749d_8eca_422a_a8e6_7d94cd3905e2.slice/crio-e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606 WatchSource:0}: Error finding container e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606: Status 404 returned error can't find the container with id e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606
Oct 30 21:58:17.006725 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:17.006693    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws" event={"ID":"e995749d-8eca-422a-a8e6-7d94cd3905e2","Type":"ContainerStarted","Data":"e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606"}
Oct 30 21:58:17.347122 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:17.347092    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"]
Oct 30 21:58:17.353796 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:17.353752    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7015b4c8_3749_45f7_bede_21dd394df0ea.slice/crio-411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32 WatchSource:0}: Error finding container 411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32: Status 404 returned error can't find the container with id 411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32
Oct 30 21:58:17.875563 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:17.873720    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:18.015009 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:18.014965    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24" event={"ID":"7015b4c8-3749-45f7-bede-21dd394df0ea","Type":"ContainerStarted","Data":"83e05121e5fe7f7efe7627d367a54bee5c8d90fd0028bc4ba472fad946c04816"}
Oct 30 21:58:18.015165 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:18.015055    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24" event={"ID":"7015b4c8-3749-45f7-bede-21dd394df0ea","Type":"ContainerStarted","Data":"411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32"}
Oct 30 21:58:18.016616 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:18.016581    5510 generic.go:356] "Generic (PLEG): container finished" podID="e995749d-8eca-422a-a8e6-7d94cd3905e2" containerID="464a438bdb8a87fe382a9cc3e37de8bb777f437c950937d6e48c79fe689216d6" exitCode=0
Oct 30 21:58:18.016725 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:18.016663    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws" event={"ID":"e995749d-8eca-422a-a8e6-7d94cd3905e2","Type":"ContainerDied","Data":"464a438bdb8a87fe382a9cc3e37de8bb777f437c950937d6e48c79fe689216d6"}
Oct 30 21:58:18.127448 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:18.127357    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24" podStartSLOduration=2.127344293 podStartE2EDuration="2.127344293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:18.124274546 +0000 UTC m=+907.108181424" watchObservedRunningTime="2026-10-30 21:58:18.127344293 +0000 UTC m=+907.111251168"
Oct 30 21:58:19.024329 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.024289    5510 generic.go:356] "Generic (PLEG): container finished" podID="7015b4c8-3749-45f7-bede-21dd394df0ea" containerID="83e05121e5fe7f7efe7627d367a54bee5c8d90fd0028bc4ba472fad946c04816" exitCode=0
Oct 30 21:58:19.024830 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.024372    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24" event={"ID":"7015b4c8-3749-45f7-bede-21dd394df0ea","Type":"ContainerDied","Data":"83e05121e5fe7f7efe7627d367a54bee5c8d90fd0028bc4ba472fad946c04816"}
Oct 30 21:58:19.190205 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.190174    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:19.246974 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.246915    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\") pod \"e995749d-8eca-422a-a8e6-7d94cd3905e2\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") "
Oct 30 21:58:19.247176 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.247134    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\" (UniqueName: \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\") pod \"e995749d-8eca-422a-a8e6-7d94cd3905e2\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") "
Oct 30 21:58:19.247258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.247196    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\") pod \"e995749d-8eca-422a-a8e6-7d94cd3905e2\" (UID: \"e995749d-8eca-422a-a8e6-7d94cd3905e2\") "
Oct 30 21:58:19.247640 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.247608    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca" pod "e995749d-8eca-422a-a8e6-7d94cd3905e2" (UID: "e995749d-8eca-422a-a8e6-7d94cd3905e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:19.249273 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.249247    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk" pod "e995749d-8eca-422a-a8e6-7d94cd3905e2" (UID: "e995749d-8eca-422a-a8e6-7d94cd3905e2"). InnerVolumeSpecName "kube-api-access-z9mnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:19.249397 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.249378    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca" pod "e995749d-8eca-422a-a8e6-7d94cd3905e2" (UID: "e995749d-8eca-422a-a8e6-7d94cd3905e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:19.348810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.348711    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mnk\" (UniqueName: \"kubernetes.io/projected/e995749d-8eca-422a-a8e6-7d94cd3905e2-kube-api-access-z9mnk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.348810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.348740    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.348810 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.348750    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e995749d-8eca-422a-a8e6-7d94cd3905e2-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.376532 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.376498    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"]
Oct 30 21:58:19.377036 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.377023    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e995749d-8eca-422a-a8e6-7d94cd3905e2" containerName="revoke-keys"
Oct 30 21:58:19.377076 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.377037    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e995749d-8eca-422a-a8e6-7d94cd3905e2" containerName="revoke-keys"
Oct 30 21:58:19.377175 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.377166    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e995749d-8eca-422a-a8e6-7d94cd3905e2" containerName="revoke-keys"
Oct 30 21:58:19.380601 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.380581    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.434822 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.434626    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"]
Oct 30 21:58:19.450366 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.450336    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85k2\" (UniqueName: \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.450500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.450385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.450500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.450480    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.551504 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.551468    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85k2\" (UniqueName: \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.551676 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.551513    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.551676 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.551540    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.552186 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.552162    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.553896 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.553845    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.604840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.604735    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85k2\" (UniqueName: \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\") pod \"maas-api-revoke-sub-35b9eb3b-nkxrb\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:19.694359 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:19.694326    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:20.043837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.040825    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws" event={"ID":"e995749d-8eca-422a-a8e6-7d94cd3905e2","Type":"ContainerDied","Data":"e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606"}
Oct 30 21:58:20.043837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.040888    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4249c1142b8149dc3fa0b24e0b415882b1033e9392401f34c86ea1484405606"
Oct 30 21:58:20.043837 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.040844    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"
Oct 30 21:58:20.094647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.094613    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"]
Oct 30 21:58:20.105187 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:20.105046    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8432a3_a4c4_4257_b454_fee41b6a5cef.slice/crio-be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817 WatchSource:0}: Error finding container be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817: Status 404 returned error can't find the container with id be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817
Oct 30 21:58:20.298350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.298274    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:20.377535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.372293    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\") pod \"7015b4c8-3749-45f7-bede-21dd394df0ea\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") "
Oct 30 21:58:20.377535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.372399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\") pod \"7015b4c8-3749-45f7-bede-21dd394df0ea\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") "
Oct 30 21:58:20.377535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.372465    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\" (UniqueName: \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\") pod \"7015b4c8-3749-45f7-bede-21dd394df0ea\" (UID: \"7015b4c8-3749-45f7-bede-21dd394df0ea\") "
Oct 30 21:58:20.377535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.373471    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca" pod "7015b4c8-3749-45f7-bede-21dd394df0ea" (UID: "7015b4c8-3749-45f7-bede-21dd394df0ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:20.379065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.378263    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc" pod "7015b4c8-3749-45f7-bede-21dd394df0ea" (UID: "7015b4c8-3749-45f7-bede-21dd394df0ea"). InnerVolumeSpecName "kube-api-access-qhgzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:20.379065 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.379432    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca" pod "7015b4c8-3749-45f7-bede-21dd394df0ea" (UID: "7015b4c8-3749-45f7-bede-21dd394df0ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:20.473841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.473810    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:20.473841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.473841    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7015b4c8-3749-45f7-bede-21dd394df0ea-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:20.474058 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.473857    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgzc\" (UniqueName: \"kubernetes.io/projected/7015b4c8-3749-45f7-bede-21dd394df0ea-kube-api-access-qhgzc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:20.931042 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.930927    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 21:58:20.931529 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.931510    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7015b4c8-3749-45f7-bede-21dd394df0ea" containerName="revoke-keys"
Oct 30 21:58:20.931566 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.931531    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7015b4c8-3749-45f7-bede-21dd394df0ea" containerName="revoke-keys"
Oct 30 21:58:20.931654 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.931645    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7015b4c8-3749-45f7-bede-21dd394df0ea" containerName="revoke-keys"
Oct 30 21:58:20.936140 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.936110    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991561    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991651    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991763    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991830    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991886    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991952    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.991992    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.992110    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvm9\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:20.999991 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:20.992190    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.249803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.239671    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.249803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.239752    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.249803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.239819    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.249803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.239864    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.249803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.239912    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.298917    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.298996    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.299073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvm9\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.299168    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.299896    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.300182    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.266929    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.314841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.300475    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.356905 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.356874    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"
Oct 30 21:58:21.375387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.375354    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.412467 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.412428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.500856 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.497099    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.526850 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.521448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.631841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.626953    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvm9\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\") pod \"maas-default-gateway-openshift-default-694db55c87-27557\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.634639 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.634606    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb" event={"ID":"ec8432a3-a4c4-4257-b454-fee41b6a5cef","Type":"ContainerStarted","Data":"4dd5492822173d9bd1f11334437277d63ae1e80367e0a9d89365054768721da5"}
Oct 30 21:58:21.634855 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.634838    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 21:58:21.635420 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.635386    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb" event={"ID":"ec8432a3-a4c4-4257-b454-fee41b6a5cef","Type":"ContainerStarted","Data":"be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817"}
Oct 30 21:58:21.635613 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.635596    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24" event={"ID":"7015b4c8-3749-45f7-bede-21dd394df0ea","Type":"ContainerDied","Data":"411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32"}
Oct 30 21:58:21.635716 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.635705    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411665073e259fb60c9723142f1213049db46053c7598f6b81ae7f5e1d884f32"
Oct 30 21:58:21.824920 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.820963    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:21.859604 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.859573    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:21.918695 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.918639    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb" podStartSLOduration=2.918619095 podStartE2EDuration="2.918619095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:21.692645101 +0000 UTC m=+910.676551978" watchObservedRunningTime="2026-10-30 21:58:21.918619095 +0000 UTC m=+910.902525972"
Oct 30 21:58:21.947034 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.946971    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 21:58:21.982824 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:21.976359    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.005208 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.005130    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 21:58:22.055569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.055528    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.163321 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.163230    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.163647 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.163622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.163945 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.163921    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.164131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.164113    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.214133 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.214024    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.222895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.216934    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.222895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.217009    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.222895 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.217074    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64b4m\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.398813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.398589    5510 generic.go:356] "Generic (PLEG): container finished" podID="ec8432a3-a4c4-4257-b454-fee41b6a5cef" containerID="4dd5492822173d9bd1f11334437277d63ae1e80367e0a9d89365054768721da5" exitCode=0
Oct 30 21:58:22.398813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.398667    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb" event={"ID":"ec8432a3-a4c4-4257-b454-fee41b6a5cef","Type":"ContainerDied","Data":"4dd5492822173d9bd1f11334437277d63ae1e80367e0a9d89365054768721da5"}
Oct 30 21:58:22.399908 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.399882    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.401234 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.401199    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.401915 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.401893    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.402898 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.402877    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.403056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.403036    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64b4m\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.408548 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.408524    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.408738 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.408719    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.408907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.408889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.409075 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.409059    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.409193 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.409177    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.409672 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.409650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.410293 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.410271    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.411408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.410719    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.411524 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.402726    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.411924 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.411905    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.435160 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.435085    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.470809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.470484    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64b4m\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.477560 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.477161    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.522531 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.522489    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.792482 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.792445    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:22.942368 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.942321    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 21:58:22.948590 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:22.948561    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c22ae2_d03d_4c08_a65b_8f866b30869b.slice/crio-8b02dc2ecd7189d0860dedfb92880fa8785072c37a9479131bbf586d0e12bcc8 WatchSource:0}: Error finding container 8b02dc2ecd7189d0860dedfb92880fa8785072c37a9479131bbf586d0e12bcc8: Status 404 returned error can't find the container with id 8b02dc2ecd7189d0860dedfb92880fa8785072c37a9479131bbf586d0e12bcc8
Oct 30 21:58:22.951047 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.951011    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:22.951158 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.951093    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:22.951158 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:22.951135    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:23.171857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.171769    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 21:58:23.179273 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:23.179232    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61efb1f7_04b2_49bc_a633_de94b399353f.slice/crio-8a56936509280911535ec036d6e79d83115037dc7bc31baeccfb90f380353d90 WatchSource:0}: Error finding container 8a56936509280911535ec036d6e79d83115037dc7bc31baeccfb90f380353d90: Status 404 returned error can't find the container with id 8a56936509280911535ec036d6e79d83115037dc7bc31baeccfb90f380353d90
Oct 30 21:58:23.182028 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.181981    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:23.182132 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.182075    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:23.182132 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.182113    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:23.407229 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.407191    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" event={"ID":"61efb1f7-04b2-49bc-a633-de94b399353f","Type":"ContainerStarted","Data":"c91010de94c646711db7df6a37cea71ec2fb4cf2916a5bba1900004381518770"}
Oct 30 21:58:23.407655 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.407249    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" event={"ID":"61efb1f7-04b2-49bc-a633-de94b399353f","Type":"ContainerStarted","Data":"8a56936509280911535ec036d6e79d83115037dc7bc31baeccfb90f380353d90"}
Oct 30 21:58:23.409183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.409157    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" event={"ID":"e9c22ae2-d03d-4c08-a65b-8f866b30869b","Type":"ContainerStarted","Data":"225bcf12356246798a18d8235c593557fb987c81fc3b0e567a850097501d4c1f"}
Oct 30 21:58:23.409312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.409202    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" event={"ID":"e9c22ae2-d03d-4c08-a65b-8f866b30869b","Type":"ContainerStarted","Data":"8b02dc2ecd7189d0860dedfb92880fa8785072c37a9479131bbf586d0e12bcc8"}
Oct 30 21:58:23.564831 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.551654    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podStartSLOduration=2.551635412 podStartE2EDuration="2.551635412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.549553953 +0000 UTC m=+912.533460830" watchObservedRunningTime="2026-10-30 21:58:23.551635412 +0000 UTC m=+912.535542290"
Oct 30 21:58:23.623662 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.623632    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:23.686093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.686032    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podStartSLOduration=3.686013672 podStartE2EDuration="3.686013672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.682331078 +0000 UTC m=+912.666237955" watchObservedRunningTime="2026-10-30 21:58:23.686013672 +0000 UTC m=+912.669920539"
Oct 30 21:58:23.754852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.742369    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\" (UniqueName: \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\") pod \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") "
Oct 30 21:58:23.754852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.742450    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\") pod \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") "
Oct 30 21:58:23.754852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.742578    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\") pod \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\" (UID: \"ec8432a3-a4c4-4257-b454-fee41b6a5cef\") "
Oct 30 21:58:23.754852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.742870    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca" pod "ec8432a3-a4c4-4257-b454-fee41b6a5cef" (UID: "ec8432a3-a4c4-4257-b454-fee41b6a5cef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:23.754852 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.743156    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.791812 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.791527    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2" pod "ec8432a3-a4c4-4257-b454-fee41b6a5cef" (UID: "ec8432a3-a4c4-4257-b454-fee41b6a5cef"). InnerVolumeSpecName "kube-api-access-x85k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:23.793270 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.793234    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:23.793450 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.793425    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca" pod "ec8432a3-a4c4-4257-b454-fee41b6a5cef" (UID: "ec8432a3-a4c4-4257-b454-fee41b6a5cef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:23.844394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.844360    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85k2\" (UniqueName: \"kubernetes.io/projected/ec8432a3-a4c4-4257-b454-fee41b6a5cef-kube-api-access-x85k2\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.844394 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.844392    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec8432a3-a4c4-4257-b454-fee41b6a5cef-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.873803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:23.860889    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:24.421616 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.421586    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"
Oct 30 21:58:24.422046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.421613    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb" event={"ID":"ec8432a3-a4c4-4257-b454-fee41b6a5cef","Type":"ContainerDied","Data":"be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817"}
Oct 30 21:58:24.422046 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.421662    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be7b1f1c49996be2f443a14eec12f31511c59c8aedbadb20d1c22ab264ad6817"
Oct 30 21:58:24.794801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.794269    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.794801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.794329    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:24.871808 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.868798    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.871808 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:24.868855    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:25.333818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.331858    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"]
Oct 30 21:58:25.333818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.332440    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ec8432a3-a4c4-4257-b454-fee41b6a5cef" containerName="revoke-keys"
Oct 30 21:58:25.333818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.332452    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8432a3-a4c4-4257-b454-fee41b6a5cef" containerName="revoke-keys"
Oct 30 21:58:25.333818 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.332538    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8432a3-a4c4-4257-b454-fee41b6a5cef" containerName="revoke-keys"
Oct 30 21:58:25.336156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.336137    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.385805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.385531    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:25.426901 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.424267    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"]
Oct 30 21:58:25.431592 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.431569    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:25.431774 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.431701    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:25.431900 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.431809    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:25.515850 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.504582    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.515850 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.504650    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.515850 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.504677    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvpk\" (UniqueName: \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.610714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.610617    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.610992 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.610969    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.611121 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.611104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvpk\" (UniqueName: \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.614600 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.614527    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.616183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.616160    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.711425 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.711393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvpk\" (UniqueName: \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\") pod \"maas-api-revoke-sub-3716a776-2vd8c\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:25.796223 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.796185    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.796531 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.796485    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:25.864859 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.863114    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.864859 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.863171    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:25.950142 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:25.950111    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:26.416756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.416057    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"]
Oct 30 21:58:26.420377 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:26.420345    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bf61af_6adf_4c07_b2b9_b22c1bd947e1.slice/crio-64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b WatchSource:0}: Error finding container 64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b: Status 404 returned error can't find the container with id 64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b
Oct 30 21:58:26.446487 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.446458    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c" event={"ID":"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1","Type":"ContainerStarted","Data":"64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b"}
Oct 30 21:58:26.793813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.792924    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.793813 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.792993    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:26.861861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.861194    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.861861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:26.861277    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:27.232147 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.232111    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:27.504859 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.489650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c" event={"ID":"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1","Type":"ContainerStarted","Data":"988ece0732c241738e1cd347566a58155b654e5d33a9e978ee974fe46597e690"}
Oct 30 21:58:27.613857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.607364    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c" podStartSLOduration=2.60734303 podStartE2EDuration="2.60734303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:27.600193128 +0000 UTC m=+916.584100006" watchObservedRunningTime="2026-10-30 21:58:27.60734303 +0000 UTC m=+916.591249907"
Oct 30 21:58:27.796631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.796107    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.796631 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.796199    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:27.865258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.865140    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.865258 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:27.865198    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:28.516131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.513211    5510 generic.go:356] "Generic (PLEG): container finished" podID="e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" containerID="988ece0732c241738e1cd347566a58155b654e5d33a9e978ee974fe46597e690" exitCode=0
Oct 30 21:58:28.516131 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.513440    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c" event={"ID":"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1","Type":"ContainerDied","Data":"988ece0732c241738e1cd347566a58155b654e5d33a9e978ee974fe46597e690"}
Oct 30 21:58:28.807862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.807366    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.807862 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.807429    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:28.870734 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.870682    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.870954 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:28.870743    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:29.703356 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.703331    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:29.719668 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.719638    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\" (UniqueName: \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\") pod \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") "
Oct 30 21:58:29.719811 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.719686    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\") pod \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") "
Oct 30 21:58:29.719864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.719841    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\") pod \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\" (UID: \"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1\") "
Oct 30 21:58:29.720149 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.720116    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca" pod "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" (UID: "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:29.720314 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.720294    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:29.721775 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.721752    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk" pod "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" (UID: "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1"). InnerVolumeSpecName "kube-api-access-fzvpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:29.722244 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.722223    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca" pod "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" (UID: "e8bf61af-6adf-4c07-b2b9-b22c1bd947e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:29.793135 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.793074    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.793272 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.793142    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:58:29.822156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.821951    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvpk\" (UniqueName: \"kubernetes.io/projected/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-kube-api-access-fzvpk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:29.822156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.821989    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:29.861449 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.861415    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-27557 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.861622 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:29.861475    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:58:30.170890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.169303    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"]
Oct 30 21:58:30.170890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.170228    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" containerName="revoke-keys"
Oct 30 21:58:30.170890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.170248    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" containerName="revoke-keys"
Oct 30 21:58:30.170890 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.170434    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" containerName="revoke-keys"
Oct 30 21:58:30.174490 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.174468    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.225979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.225943    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gq5\" (UniqueName: \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.226120 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.225992    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.226172 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.226158    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.234608 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.234568    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"]
Oct 30 21:58:30.327655 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.327619    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.327857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.327737    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gq5\" (UniqueName: \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.327857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.327797    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.328474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.328448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.330039 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.330013    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.381098 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.381073    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gq5\" (UniqueName: \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\") pod \"maas-api-revoke-sub-7632c2d8-x42n7\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.495135 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.495028    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:30.532814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.531915    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c" event={"ID":"e8bf61af-6adf-4c07-b2b9-b22c1bd947e1","Type":"ContainerDied","Data":"64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b"}
Oct 30 21:58:30.532814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.531982    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f81b0bbbd751ab23f9a44921ca43e17e3a2af026010e29d74a322a078f1f9b"
Oct 30 21:58:30.532814 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.532052    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"
Oct 30 21:58:30.627195 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.627153    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"]
Oct 30 21:58:30.633177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.633152    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.680254 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.680227    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"]
Oct 30 21:58:30.732615 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.732580    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.732966 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.732758    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzm4h\" (UniqueName: \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.732966 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.732869    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.798979 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.798898    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:30.799820 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.799774    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:30.800477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.800459    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 21:58:30.834201 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.834170    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.834343 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.834326    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzm4h\" (UniqueName: \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.834408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.834391    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.834859 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.834837    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.836868 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.836846    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.887756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.887710    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"]
Oct 30 21:58:30.887989 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.887846    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzm4h\" (UniqueName: \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\") pod \"maas-api-revoke-sub-a1dd44a5-4hxkw\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:30.892949 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:30.892919    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22ad7ae_4a49_429c_9063_a2256cac5d45.slice/crio-78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e WatchSource:0}: Error finding container 78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e: Status 404 returned error can't find the container with id 78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e
Oct 30 21:58:30.947654 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:30.947630    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:31.461861 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.461825    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"]
Oct 30 21:58:31.468607 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:31.468576    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31df64b2_218c_4d93_9ccd_17c2ca0ebb94.slice/crio-8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854 WatchSource:0}: Error finding container 8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854: Status 404 returned error can't find the container with id 8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854
Oct 30 21:58:31.541144 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.541102    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw" event={"ID":"31df64b2-218c-4d93-9ccd-17c2ca0ebb94","Type":"ContainerStarted","Data":"8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854"}
Oct 30 21:58:31.542962 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.542934    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7" event={"ID":"c22ad7ae-4a49-429c-9063-a2256cac5d45","Type":"ContainerStarted","Data":"030e8105ebbebe0e2ce5570d1a1a7b8a1fbd51fbcdc14d279072bd9c91764d34"}
Oct 30 21:58:31.543077 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.542974    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7" event={"ID":"c22ad7ae-4a49-429c-9063-a2256cac5d45","Type":"ContainerStarted","Data":"78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e"}
Oct 30 21:58:31.625321 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.625290    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:31.625756 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.625734    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:31.626708 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.626691    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 21:58:31.650769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.650726    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7" podStartSLOduration=1.6507150259999999 podStartE2EDuration="1.650715026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:31.649983723 +0000 UTC m=+920.633890599" watchObservedRunningTime="2026-10-30 21:58:31.650715026 +0000 UTC m=+920.634621932"
Oct 30 21:58:31.785817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:31.785612    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:32.550268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:32.550232    5510 generic.go:356] "Generic (PLEG): container finished" podID="31df64b2-218c-4d93-9ccd-17c2ca0ebb94" containerID="feb13e84a1762b70f0616f35bb7b64ce72d36e45b3062e5a6319281bf9c85eeb" exitCode=0
Oct 30 21:58:32.550446 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:32.550315    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw" event={"ID":"31df64b2-218c-4d93-9ccd-17c2ca0ebb94","Type":"ContainerDied","Data":"feb13e84a1762b70f0616f35bb7b64ce72d36e45b3062e5a6319281bf9c85eeb"}
Oct 30 21:58:32.551970 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:32.551946    5510 generic.go:356] "Generic (PLEG): container finished" podID="c22ad7ae-4a49-429c-9063-a2256cac5d45" containerID="030e8105ebbebe0e2ce5570d1a1a7b8a1fbd51fbcdc14d279072bd9c91764d34" exitCode=0
Oct 30 21:58:32.552096 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:32.552015    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7" event={"ID":"c22ad7ae-4a49-429c-9063-a2256cac5d45","Type":"ContainerDied","Data":"030e8105ebbebe0e2ce5570d1a1a7b8a1fbd51fbcdc14d279072bd9c91764d34"}
Oct 30 21:58:33.741491 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.741439    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:33.745187 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.745168    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:33.784553 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784527    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\") pod \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") "
Oct 30 21:58:33.784658 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784605    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\") pod \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") "
Oct 30 21:58:33.784658 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784626    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\" (UniqueName: \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\") pod \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\" (UID: \"31df64b2-218c-4d93-9ccd-17c2ca0ebb94\") "
Oct 30 21:58:33.784801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784684    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\") pod \"c22ad7ae-4a49-429c-9063-a2256cac5d45\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") "
Oct 30 21:58:33.784801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784715    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\") pod \"c22ad7ae-4a49-429c-9063-a2256cac5d45\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") "
Oct 30 21:58:33.784801 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.784754    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\" (UniqueName: \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\") pod \"c22ad7ae-4a49-429c-9063-a2256cac5d45\" (UID: \"c22ad7ae-4a49-429c-9063-a2256cac5d45\") "
Oct 30 21:58:33.785094 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.785067    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca" pod "c22ad7ae-4a49-429c-9063-a2256cac5d45" (UID: "c22ad7ae-4a49-429c-9063-a2256cac5d45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:33.785268 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.785243    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca" pod "31df64b2-218c-4d93-9ccd-17c2ca0ebb94" (UID: "31df64b2-218c-4d93-9ccd-17c2ca0ebb94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:33.785376 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.785293    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.786860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.786763    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca" pod "c22ad7ae-4a49-429c-9063-a2256cac5d45" (UID: "c22ad7ae-4a49-429c-9063-a2256cac5d45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:33.786966 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.786947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5" pod "c22ad7ae-4a49-429c-9063-a2256cac5d45" (UID: "c22ad7ae-4a49-429c-9063-a2256cac5d45"). InnerVolumeSpecName "kube-api-access-d9gq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:33.787042 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.786979    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h" pod "31df64b2-218c-4d93-9ccd-17c2ca0ebb94" (UID: "31df64b2-218c-4d93-9ccd-17c2ca0ebb94"). InnerVolumeSpecName "kube-api-access-bzm4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:33.787093 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.787040    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca" pod "31df64b2-218c-4d93-9ccd-17c2ca0ebb94" (UID: "31df64b2-218c-4d93-9ccd-17c2ca0ebb94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:33.886722 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.886673    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.886941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.886744    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.886941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.886762    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzm4h\" (UniqueName: \"kubernetes.io/projected/31df64b2-218c-4d93-9ccd-17c2ca0ebb94-kube-api-access-bzm4h\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.886941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.886791    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22ad7ae-4a49-429c-9063-a2256cac5d45-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.886941 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:33.886806    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9gq5\" (UniqueName: \"kubernetes.io/projected/c22ad7ae-4a49-429c-9063-a2256cac5d45-kube-api-access-d9gq5\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:34.565841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.565811    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"
Oct 30 21:58:34.565841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.565823    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw" event={"ID":"31df64b2-218c-4d93-9ccd-17c2ca0ebb94","Type":"ContainerDied","Data":"8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854"}
Oct 30 21:58:34.566111 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.565871    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb4849dc05a87cea4ba576850832066b1522079a8ee85b102c5e4488458c854"
Oct 30 21:58:34.567221 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.567205    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"
Oct 30 21:58:34.567320 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.567221    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7" event={"ID":"c22ad7ae-4a49-429c-9063-a2256cac5d45","Type":"ContainerDied","Data":"78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e"}
Oct 30 21:58:34.567320 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:34.567252    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78079fb792725cf488e5126cd7b25345301ad7fb96b87aef555a091bec15043e"
Oct 30 21:58:36.192181 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:36.192143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:38.440242 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.440208    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 21:58:38.440860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.440843    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c22ad7ae-4a49-429c-9063-a2256cac5d45" containerName="revoke-keys"
Oct 30 21:58:38.440860 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.440860    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22ad7ae-4a49-429c-9063-a2256cac5d45" containerName="revoke-keys"
Oct 30 21:58:38.440965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.440872    5510 cpu_manager.go:414] "containerMap: removing container" podUID="31df64b2-218c-4d93-9ccd-17c2ca0ebb94" containerName="revoke-keys"
Oct 30 21:58:38.440965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.440877    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31df64b2-218c-4d93-9ccd-17c2ca0ebb94" containerName="revoke-keys"
Oct 30 21:58:38.441038 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.441013    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22ad7ae-4a49-429c-9063-a2256cac5d45" containerName="revoke-keys"
Oct 30 21:58:38.441038 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.441024    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="31df64b2-218c-4d93-9ccd-17c2ca0ebb94" containerName="revoke-keys"
Oct 30 21:58:38.445760 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.445740    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.486247 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.486222    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 21:58:38.489994 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.489974    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.533419 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.533391    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544094    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfhp\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544161    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544200    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544247    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544302    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544332    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrztk\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544371    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544404    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544465    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544518    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544604    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544650    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544695    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544720    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544773    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544833    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.544882    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.546344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.546274    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 21:58:38.645749 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645711    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.645938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645760    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfhp\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.645938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645841    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.645938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645875    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.645938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645918    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646202 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645940    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646202 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.645971    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646202 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646132    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrztk\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646202 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646188    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646232    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646300    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646314    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646357    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646413    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646455    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646498    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646544    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646581    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646631    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646749    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.646809 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646746    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.647207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646823    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.647207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.646996    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.647207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.647121    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.647207 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.647127    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.648580 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.648554    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.648714 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.648697    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.649022 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.649001    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.649234 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.649215    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.699339 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.699274    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.699769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.699745    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.699883 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.699827    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrztk\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.701350 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.701325    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfhp\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.767992 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.767963    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:38.776322 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.776288    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 21:58:38.783600 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.783579    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.802732 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.802709    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.847863    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849585    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849646    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849669    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849703    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlwb\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849747    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849765    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849880    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.849929    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.853914 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.850001    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951701 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951635    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951701 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951674    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951725    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlwb\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951807    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951838    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.951944 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951915    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952150 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.951961    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952150 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952041    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952150 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952116    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952150 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952146    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952355 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952214    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952355 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952281    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952525 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952507    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.952765 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.952746    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.954299 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.954279    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:38.954481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:38.954464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:39.006228 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.006196    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlwb\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:39.006916 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.006898    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:39.104389 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.104358    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:39.214916 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.214881    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 21:58:39.217358 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.217331    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 21:58:39.219539 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:39.219512    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b23a6f_a9dc_4ac5_a7ba_159ea4ddd7e7.slice/crio-bb23b64a15c152de1b3f6fefb6dc21c57911051cf6cb56eb672d6a054709ebce WatchSource:0}: Error finding container bb23b64a15c152de1b3f6fefb6dc21c57911051cf6cb56eb672d6a054709ebce: Status 404 returned error can't find the container with id bb23b64a15c152de1b3f6fefb6dc21c57911051cf6cb56eb672d6a054709ebce
Oct 30 21:58:39.221320 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:39.221288    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8ed86c_a90b_4fce_9c19_e6de582c2c2a.slice/crio-62f7872c9508374ab4a4b4f5343804d58362c59663dbef2775ad276757965285 WatchSource:0}: Error finding container 62f7872c9508374ab4a4b4f5343804d58362c59663dbef2775ad276757965285: Status 404 returned error can't find the container with id 62f7872c9508374ab4a4b4f5343804d58362c59663dbef2775ad276757965285
Oct 30 21:58:39.223477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.223249    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.223477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.223326    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.223477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.223370    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.223950 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.223918    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.224015 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.223997    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.224056 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.224044    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.489104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.489022    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 21:58:39.494206 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:39.494179    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188c5dac_d2c7_4465_89c0_6d77551d9234.slice/crio-9133a79dd53e729ef3637a1b292769587d6d36a95bbe9119acccc64caa8320de WatchSource:0}: Error finding container 9133a79dd53e729ef3637a1b292769587d6d36a95bbe9119acccc64caa8320de: Status 404 returned error can't find the container with id 9133a79dd53e729ef3637a1b292769587d6d36a95bbe9119acccc64caa8320de
Oct 30 21:58:39.497580 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.497541    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.497694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.497627    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.497694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.497669    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:58:39.604346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.604313    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" event={"ID":"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a","Type":"ContainerStarted","Data":"eddff01c2da4c556f9e073e539c2c55f272284188f4d4573aa2310f2733df4b7"}
Oct 30 21:58:39.605372 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.605334    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" event={"ID":"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a","Type":"ContainerStarted","Data":"62f7872c9508374ab4a4b4f5343804d58362c59663dbef2775ad276757965285"}
Oct 30 21:58:39.606736 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.606483    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" event={"ID":"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7","Type":"ContainerStarted","Data":"3af6224a9a688d5f3efc6e5afcce08bbe0c9ccffdd1b6f754a540f96424aa101"}
Oct 30 21:58:39.606736 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.606529    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" event={"ID":"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7","Type":"ContainerStarted","Data":"bb23b64a15c152de1b3f6fefb6dc21c57911051cf6cb56eb672d6a054709ebce"}
Oct 30 21:58:39.608571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.608490    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" event={"ID":"188c5dac-d2c7-4465-89c0-6d77551d9234","Type":"ContainerStarted","Data":"1105d08b382fd1dafa834dcfab09673325b0c624c6c642e6c57445df616a804c"}
Oct 30 21:58:39.608571 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.608535    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" event={"ID":"188c5dac-d2c7-4465-89c0-6d77551d9234","Type":"ContainerStarted","Data":"9133a79dd53e729ef3637a1b292769587d6d36a95bbe9119acccc64caa8320de"}
Oct 30 21:58:39.768658 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.768574    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:39.803815 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.803546    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:39.833832 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.833264    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podStartSLOduration=1.833245621 podStartE2EDuration="1.833245621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.831884663 +0000 UTC m=+928.815791542" watchObservedRunningTime="2026-10-30 21:58:39.833245621 +0000 UTC m=+928.817152498"
Oct 30 21:58:39.856805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.853376    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podStartSLOduration=1.8533532799999999 podStartE2EDuration="1.85335328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.716025217 +0000 UTC m=+928.699932094" watchObservedRunningTime="2026-10-30 21:58:39.85335328 +0000 UTC m=+928.837260157"
Oct 30 21:58:39.951921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:39.951859    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podStartSLOduration=1.951839144 podStartE2EDuration="1.951839144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.95112895 +0000 UTC m=+928.935035828" watchObservedRunningTime="2026-10-30 21:58:39.951839144 +0000 UTC m=+928.935746021"
Oct 30 21:58:40.108863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.107991    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:40.108863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.108053    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:40.108863 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.108323    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:40.771699 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.771659    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:40.772308 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.772277    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:40.818106 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.818068    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:40.818387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.818360    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:40.995045 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:40.994999    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:41.108163 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.108077    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:41.108432 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.108405    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:41.592808 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.590547    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"]
Oct 30 21:58:41.603843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.603041    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.642025 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.641604    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"]
Oct 30 21:58:41.697577 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.697539    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:41.712906 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.698022    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:41.713141 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.698726    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:41.713287 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.703898    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:41.739907 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.739875    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.740189 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.740167    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.740419 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.740402    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrnl\" (UniqueName: \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.799260 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.799223    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:41.803889 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.803846    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:41.832477 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.832441    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:41.832723 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.832693    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:41.841840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.841730    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.841840 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.841812    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.842333 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.842310    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrnl\" (UniqueName: \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.842750 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.842711    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.845805 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.845655    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.945880 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.945841    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrnl\" (UniqueName: \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\") pod \"maas-api-revoke-sub-af7175e3-tj6pm\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:41.946075 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:41.945994    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:42.117864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.116891    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.117864 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.116970    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:42.850156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.849911    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.850156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.849968    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:42.889841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.885529    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.889841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:42.885617    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:43.067047 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.065599    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"]
Oct 30 21:58:43.107190 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:43.106899    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263488a1_3a03_4a71_adf5_1817370769b3.slice/crio-6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a WatchSource:0}: Error finding container 6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a: Status 404 returned error can't find the container with id 6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a
Oct 30 21:58:43.107431 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.107407    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.110990 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.110959    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:43.549060 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:58:43.545365    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:58:43.779819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.769585    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.779819 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.769649    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:43.805147 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.805112    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.805384 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.805356    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:43.942217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.941148    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm" event={"ID":"263488a1-3a03-4a71-adf5-1817370769b3","Type":"ContainerStarted","Data":"a6fb707cbe067994cfb7bf7c3a8aa82756bc0a1dd163e295c04b1e016b9f0d50"}
Oct 30 21:58:43.942217 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:43.941219    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm" event={"ID":"263488a1-3a03-4a71-adf5-1817370769b3","Type":"ContainerStarted","Data":"6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a"}
Oct 30 21:58:44.091865 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.072463    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm" podStartSLOduration=3.072442817 podStartE2EDuration="3.072442817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:44.069653623 +0000 UTC m=+933.053560501" watchObservedRunningTime="2026-10-30 21:58:44.072442817 +0000 UTC m=+933.056349694"
Oct 30 21:58:44.148893 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.110284    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.155072 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.154886    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:44.776804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.776183    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.776804 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.776255    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:44.829720 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.829327    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.829720 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.829390    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:44.951347 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.951310    5510 generic.go:356] "Generic (PLEG): container finished" podID="263488a1-3a03-4a71-adf5-1817370769b3" containerID="a6fb707cbe067994cfb7bf7c3a8aa82756bc0a1dd163e295c04b1e016b9f0d50" exitCode=0
Oct 30 21:58:44.951740 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:44.951440    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm" event={"ID":"263488a1-3a03-4a71-adf5-1817370769b3","Type":"ContainerDied","Data":"a6fb707cbe067994cfb7bf7c3a8aa82756bc0a1dd163e295c04b1e016b9f0d50"}
Oct 30 21:58:45.110634 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.110537    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.110931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.110882    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:45.795263 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.795221    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.795556 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.795523    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:58:45.803222 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.803176    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.803457 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:45.803432    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:58:46.106875 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.105220    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:58:46.106875 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.105357    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:58:46.166799 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.166760    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:46.303574 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.302337    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\") pod \"263488a1-3a03-4a71-adf5-1817370769b3\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") "
Oct 30 21:58:46.303574 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.302475    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\") pod \"263488a1-3a03-4a71-adf5-1817370769b3\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") "
Oct 30 21:58:46.303574 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.302576    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\" (UniqueName: \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\") pod \"263488a1-3a03-4a71-adf5-1817370769b3\" (UID: \"263488a1-3a03-4a71-adf5-1817370769b3\") "
Oct 30 21:58:46.328807 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.321338    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl" pod "263488a1-3a03-4a71-adf5-1817370769b3" (UID: "263488a1-3a03-4a71-adf5-1817370769b3"). InnerVolumeSpecName "kube-api-access-vbrnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:46.330434 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.330398    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca" pod "263488a1-3a03-4a71-adf5-1817370769b3" (UID: "263488a1-3a03-4a71-adf5-1817370769b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:46.334099 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.334052    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca" pod "263488a1-3a03-4a71-adf5-1817370769b3" (UID: "263488a1-3a03-4a71-adf5-1817370769b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:46.411701 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.405984    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:46.438058 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.412390    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:46.438934 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.438911    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrnl\" (UniqueName: \"kubernetes.io/projected/263488a1-3a03-4a71-adf5-1817370769b3-kube-api-access-vbrnl\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:46.439145 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.439126    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/263488a1-3a03-4a71-adf5-1817370769b3-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:46.772694 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.772662    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:46.773133 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.773104    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:46.773769 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.773748    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 21:58:46.828398 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.827461    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:46.828398 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.828069    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:46.828742 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.828722    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 21:58:46.960370 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.960334    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:58:46.969266 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.969238    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"
Oct 30 21:58:46.969412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.969290    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm" event={"ID":"263488a1-3a03-4a71-adf5-1817370769b3","Type":"ContainerDied","Data":"6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a"}
Oct 30 21:58:46.969412 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:46.969332    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0e3aa0406a3c781d12278b4da9b61b11754816f4931a3be2edb73f83d6be3a"
Oct 30 21:58:47.159132 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:47.159044    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:47.159639 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:47.159622    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:47.160654 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:47.160635    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 21:58:50.932967 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:50.932934    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:51.800931 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.800896    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"]
Oct 30 21:58:51.801499 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.801484    5510 cpu_manager.go:414] "containerMap: removing container" podUID="263488a1-3a03-4a71-adf5-1817370769b3" containerName="revoke-keys"
Oct 30 21:58:51.801592 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.801499    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263488a1-3a03-4a71-adf5-1817370769b3" containerName="revoke-keys"
Oct 30 21:58:51.801592 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.801579    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="263488a1-3a03-4a71-adf5-1817370769b3" containerName="revoke-keys"
Oct 30 21:58:51.806132 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.806112    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:51.856328 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.856296    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:51.856506 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.856428    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:58:51.901535 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.901497    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"]
Oct 30 21:58:51.902698 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.902675    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:51.902886 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.902711    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:51.946405 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.946378    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:51.946754 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.946419    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:51.946754 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:51.946613    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtz96\" (UniqueName: \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.047378 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.047348    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.047569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.047388    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.047569 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.047491    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtz96\" (UniqueName: \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.048104 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.048078    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.049827 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.049808    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.100691 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.100628    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtz96\" (UniqueName: \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\") pod \"maas-api-revoke-sub-b495d828-bhxwt\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.119354 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.119325    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:52.494341 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:52.494296    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"]
Oct 30 21:58:52.501918 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:58:52.501890    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548e4216_37f1_49c3_a1fe_666294e32e14.slice/crio-278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a WatchSource:0}: Error finding container 278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a: Status 404 returned error can't find the container with id 278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a
Oct 30 21:58:53.028005 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:53.027920    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt" event={"ID":"548e4216-37f1-49c3-a1fe-666294e32e14","Type":"ContainerStarted","Data":"9e455e7a5495d02ee3eedbd887ba8d13aa85fdacebd77eddead6f0b4d08242ec"}
Oct 30 21:58:53.028005 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:53.027974    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt" event={"ID":"548e4216-37f1-49c3-a1fe-666294e32e14","Type":"ContainerStarted","Data":"278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a"}
Oct 30 21:58:53.135388 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:53.135337    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt" podStartSLOduration=2.135322929 podStartE2EDuration="2.135322929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:53.133820621 +0000 UTC m=+942.117727497" watchObservedRunningTime="2026-10-30 21:58:53.135322929 +0000 UTC m=+942.119229804"
Oct 30 21:58:54.035534 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:54.035496    5510 generic.go:356] "Generic (PLEG): container finished" podID="548e4216-37f1-49c3-a1fe-666294e32e14" containerID="9e455e7a5495d02ee3eedbd887ba8d13aa85fdacebd77eddead6f0b4d08242ec" exitCode=0
Oct 30 21:58:54.036041 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:54.035579    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt" event={"ID":"548e4216-37f1-49c3-a1fe-666294e32e14","Type":"ContainerDied","Data":"9e455e7a5495d02ee3eedbd887ba8d13aa85fdacebd77eddead6f0b4d08242ec"}
Oct 30 21:58:55.189157 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.189126    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:55.288017 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.287983    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\") pod \"548e4216-37f1-49c3-a1fe-666294e32e14\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") "
Oct 30 21:58:55.288185 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.288035    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\" (UniqueName: \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\") pod \"548e4216-37f1-49c3-a1fe-666294e32e14\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") "
Oct 30 21:58:55.288185 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.288132    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\") pod \"548e4216-37f1-49c3-a1fe-666294e32e14\" (UID: \"548e4216-37f1-49c3-a1fe-666294e32e14\") "
Oct 30 21:58:55.288334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.288311    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca" pod "548e4216-37f1-49c3-a1fe-666294e32e14" (UID: "548e4216-37f1-49c3-a1fe-666294e32e14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:55.288626 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.288607    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:55.290044 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.290025    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca" pod "548e4216-37f1-49c3-a1fe-666294e32e14" (UID: "548e4216-37f1-49c3-a1fe-666294e32e14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:55.290183 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.290162    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96" pod "548e4216-37f1-49c3-a1fe-666294e32e14" (UID: "548e4216-37f1-49c3-a1fe-666294e32e14"). InnerVolumeSpecName "kube-api-access-vtz96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:55.390216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.390185    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtz96\" (UniqueName: \"kubernetes.io/projected/548e4216-37f1-49c3-a1fe-666294e32e14-kube-api-access-vtz96\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:55.390216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.390213    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548e4216-37f1-49c3-a1fe-666294e32e14-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:55.455620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.455583    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:55.880790 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:55.880749    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:56.050271 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:56.050245    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"
Oct 30 21:58:56.050435 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:56.050283    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt" event={"ID":"548e4216-37f1-49c3-a1fe-666294e32e14","Type":"ContainerDied","Data":"278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a"}
Oct 30 21:58:56.050435 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:56.050327    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278c09573f94ea97853d6df232d9ea6adecd70e22288dba5cbd1a76b6cafac0a"
Oct 30 21:58:56.606400 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:56.606368    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:59.906957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:58:59.906922    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:01.950004 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:01.949969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:04.304695 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:04.304663    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:07.524670 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.524638    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"]
Oct 30 21:59:07.525331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.525312    5510 cpu_manager.go:414] "containerMap: removing container" podUID="548e4216-37f1-49c3-a1fe-666294e32e14" containerName="revoke-keys"
Oct 30 21:59:07.525331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.525331    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548e4216-37f1-49c3-a1fe-666294e32e14" containerName="revoke-keys"
Oct 30 21:59:07.525506 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.525426    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="548e4216-37f1-49c3-a1fe-666294e32e14" containerName="revoke-keys"
Oct 30 21:59:07.529538 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.529519    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.571315 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.571276    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 21:59:07.575500 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.575482    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.580565 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.580538    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:07.585857 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.585830    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"]
Oct 30 21:59:07.586481 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.586455    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:07.594387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.594363    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 21:59:07.613357 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.613330    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:59:07.613498 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.613375    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:59:07.654457 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654425    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.654684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654662    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.654767 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654751    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.654851 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654833    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj65p\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.654910 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654870    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9glj\" (UniqueName: \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.654965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654915    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.654965 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654946    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.655122 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.654997    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.655280 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.655262    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.655346 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.655293    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.655405 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.655366    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.655461 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.655410    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.677085 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.677053    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 21:59:07.742588 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.742557    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:07.756587 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756555    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.756730 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756605    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.756730 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756652    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.756854 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756747    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.756854 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756772    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.756961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756875    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.756961 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756917    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.757074 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.756976    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757074 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757044    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757077    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757125    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj65p\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757159    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9glj\" (UniqueName: \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.757331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757178    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757331 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757227    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757485 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757465    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757893    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.757921 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757911    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.758138 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.757908    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.759675 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.759650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.759759 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.759669    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.759938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.759922    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.810797 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.810707    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.811197 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.811171    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj65p\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:07.811312 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.811294    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9glj\" (UniqueName: \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\") pod \"maas-api-revoke-sub-9e56100e-gpwnv\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.843334 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.843297    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:07.890206 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:07.890169    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:08.222368 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.222334    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"]
Oct 30 21:59:08.228050 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:59:08.228019    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705452ae_6e58_46ea_877b_2ac7d498071c.slice/crio-f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47 WatchSource:0}: Error finding container f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47: Status 404 returned error can't find the container with id f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47
Oct 30 21:59:08.338824 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.338757    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 21:59:08.344698 ip-10-0-131-231 kubenswrapper[5510]: W1030 21:59:08.344669    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537b4002_4472_4fce_8e88_bd52bf63a5e9.slice/crio-7f68d871f32e27031a87617414d561947ab29f3461f64003b5e2aa8b18c4e623 WatchSource:0}: Error finding container 7f68d871f32e27031a87617414d561947ab29f3461f64003b5e2aa8b18c4e623: Status 404 returned error can't find the container with id 7f68d871f32e27031a87617414d561947ab29f3461f64003b5e2aa8b18c4e623
Oct 30 21:59:08.347216 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.347184    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:08.347292 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.347254    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:08.347292 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.347285    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 21:59:08.725361 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:08.725285    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:09.145579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.145481    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv" event={"ID":"705452ae-6e58-46ea-877b-2ac7d498071c","Type":"ContainerStarted","Data":"e9a0a1ef6ea925f98205b7860d3e9d234b390118d044dc4ebf5612eee78e9de5"}
Oct 30 21:59:09.145579 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.145540    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv" event={"ID":"705452ae-6e58-46ea-877b-2ac7d498071c","Type":"ContainerStarted","Data":"f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47"}
Oct 30 21:59:09.147319 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.147290    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" event={"ID":"537b4002-4472-4fce-8e88-bd52bf63a5e9","Type":"ContainerStarted","Data":"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"}
Oct 30 21:59:09.147447 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.147333    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" event={"ID":"537b4002-4472-4fce-8e88-bd52bf63a5e9","Type":"ContainerStarted","Data":"7f68d871f32e27031a87617414d561947ab29f3461f64003b5e2aa8b18c4e623"}
Oct 30 21:59:09.260803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.259379    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv" podStartSLOduration=2.259361975 podStartE2EDuration="2.259361975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:09.258145712 +0000 UTC m=+958.242052588" watchObservedRunningTime="2026-10-30 21:59:09.259361975 +0000 UTC m=+958.243268869"
Oct 30 21:59:09.389531 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.389483    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podStartSLOduration=2.389470872 podStartE2EDuration="2.389470872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:09.388134064 +0000 UTC m=+958.372040939" watchObservedRunningTime="2026-10-30 21:59:09.389470872 +0000 UTC m=+958.373377747"
Oct 30 21:59:09.899806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.898186    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:09.899806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.898348    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:09.899806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:09.898390    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:10.179881 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:10.177233    5510 generic.go:356] "Generic (PLEG): container finished" podID="705452ae-6e58-46ea-877b-2ac7d498071c" containerID="e9a0a1ef6ea925f98205b7860d3e9d234b390118d044dc4ebf5612eee78e9de5" exitCode=0
Oct 30 21:59:10.179881 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:10.178478    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv" event={"ID":"705452ae-6e58-46ea-877b-2ac7d498071c","Type":"ContainerDied","Data":"e9a0a1ef6ea925f98205b7860d3e9d234b390118d044dc4ebf5612eee78e9de5"}
Oct 30 21:59:10.890927 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:10.890893    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:10.891203 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:10.890950    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:11.383828 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.383794    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:11.444985 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.444901    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\" (UniqueName: \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\") pod \"705452ae-6e58-46ea-877b-2ac7d498071c\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") "
Oct 30 21:59:11.444985 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.444947    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\") pod \"705452ae-6e58-46ea-877b-2ac7d498071c\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") "
Oct 30 21:59:11.445178 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.445055    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\") pod \"705452ae-6e58-46ea-877b-2ac7d498071c\" (UID: \"705452ae-6e58-46ea-877b-2ac7d498071c\") "
Oct 30 21:59:11.445709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.445681    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca" pod "705452ae-6e58-46ea-877b-2ac7d498071c" (UID: "705452ae-6e58-46ea-877b-2ac7d498071c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:11.447588 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.447562    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca" pod "705452ae-6e58-46ea-877b-2ac7d498071c" (UID: "705452ae-6e58-46ea-877b-2ac7d498071c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:11.447696 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.447564    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj" pod "705452ae-6e58-46ea-877b-2ac7d498071c" (UID: "705452ae-6e58-46ea-877b-2ac7d498071c"). InnerVolumeSpecName "kube-api-access-d9glj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:11.546794 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.546744    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.546957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.546802    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9glj\" (UniqueName: \"kubernetes.io/projected/705452ae-6e58-46ea-877b-2ac7d498071c-kube-api-access-d9glj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.546957 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.546816    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705452ae-6e58-46ea-877b-2ac7d498071c-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.891376 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.891335    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:11.891558 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:11.891394    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:12.194663 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:12.194587    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"
Oct 30 21:59:12.194832 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:12.194595    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv" event={"ID":"705452ae-6e58-46ea-877b-2ac7d498071c","Type":"ContainerDied","Data":"f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47"}
Oct 30 21:59:12.194832 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:12.194702    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ad5c91f96df9407c3fadee1bf96c487368bfe14b8bc2bcb8368e22ef6b4b47"
Oct 30 21:59:12.891529 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:12.891485    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:12.891946 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:12.891551    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:13.233177 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:13.233137    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:13.361005 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:59:13.360831    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:59:13.891398 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:13.891364    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:13.891586 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:13.891420    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:14.038075 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:14.038046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:14.891344 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:14.891295    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:59:14.891556 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:14.891360    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:59:15.983432 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:15.983398    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:15.983938 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:15.983733    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:15.984597 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:15.984578    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 21:59:17.719981 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:17.719938    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:22.131178 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:22.131145    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:22.289147 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:22.289115    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:25.887646 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:25.887605    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:26.626739 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:26.626704    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.137843 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.137805    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:59:29.349835 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.349791    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 21:59:29.350474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.350452    5510 cpu_manager.go:414] "containerMap: removing container" podUID="705452ae-6e58-46ea-877b-2ac7d498071c" containerName="revoke-keys"
Oct 30 21:59:29.350474 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.350473    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705452ae-6e58-46ea-877b-2ac7d498071c" containerName="revoke-keys"
Oct 30 21:59:29.350693 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.350679    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="705452ae-6e58-46ea-877b-2ac7d498071c" containerName="revoke-keys"
Oct 30 21:59:29.354587 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.354565    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.404841 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.404145    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"distinct-w5-5402dd-kserve-self-signed-certs\""
Oct 30 21:59:29.430709 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.430679    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.430878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.430731    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.430878 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.430813    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stjd\" (UniqueName: \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.430963 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.430916    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.430998 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.430986    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.431030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.431015    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.455162 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.455133    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 21:59:29.484435 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.484401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544544    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544617    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544745    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544823    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544856    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stjd\" (UniqueName: \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.546684 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.544937    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.550765 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.550725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.560156 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.560116    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.560654 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.560630    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.566806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.561366    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.578806 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.576444    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.645803 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.643193    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stjd\" (UniqueName: \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\") pod \"distinct-w5-5402dd-kserve-84f596b5bc-6z98q\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:29.679556 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:29.679456    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:30.174620 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:30.174583    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 21:59:30.345387 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:30.345346    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" event={"ID":"2c12bd71-df32-420c-939a-31315e6b669d","Type":"ContainerStarted","Data":"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"}
Oct 30 21:59:30.345550 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:30.345431    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" event={"ID":"2c12bd71-df32-420c-939a-31315e6b669d","Type":"ContainerStarted","Data":"b3a2ddeda10f265ba9b7bb14c0e246117104d4689c21e9b082b0812ff6e2213b"}
Oct 30 21:59:30.458943 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:30.458868    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podStartSLOduration=1.458823896 podStartE2EDuration="1.458823896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.453013627 +0000 UTC m=+979.436920503" watchObservedRunningTime="2026-10-30 21:59:30.458823896 +0000 UTC m=+979.442730772"
Oct 30 21:59:30.684817 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:30.683875    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:31.218030 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:31.218002    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:35.727134 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:35.727096    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:39.680408 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:39.680376    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:39.693759 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:39.693730    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:40.209045 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:40.209015    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:40.427605 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:40.427573    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 21:59:43.353112 ip-10-0-131-231 kubenswrapper[5510]: E1030 21:59:43.353037    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 21:59:44.721000 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:44.720957    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:49.232362 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:49.232307    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.462955 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:53.462873    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.724371 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:53.724292    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:55.806888 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:55.806855    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 21:59:58.244457 ip-10-0-131-231 kubenswrapper[5510]: I1030 21:59:58.244426    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:00.186699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.186663    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:00.190975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.190951    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:00.191125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.191063    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.194966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.194946    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:00.195088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.195060    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.198615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.198594    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.232598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.232568    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:00.236340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.236320    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:00.236462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.236420    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.239422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.239402    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.239541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.239439    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:00:00.240090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.240069    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.275679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.275656    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.280154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280124    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hb7\" (UniqueName: \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.280271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280184    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.280340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280278    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wsvc\" (UniqueName: \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.280397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.280456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280441    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:00.280505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280490    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.280554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280527    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdm6\" (UniqueName: \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.280628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280600    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgsp\" (UniqueName: \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.280688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280644    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq446\" (UniqueName: \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.280847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280823    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.280949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.280865    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.284138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.284116    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:00.290311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.290290    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:00.333566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.333536    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:00.371918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.371891    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:00.382307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382280    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.382468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382320    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.382468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382365    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hb7\" (UniqueName: \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.382468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382401    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.382468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382432    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wsvc\" (UniqueName: \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.382686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382471    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.382686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382613    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.382686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382659    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdm6\" (UniqueName: \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.382858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382701    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgsp\" (UniqueName: \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.382858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.382730    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq446\" (UniqueName: \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.383184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.383158    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.383276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.383159    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.383704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.383681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.383839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.383816    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.384104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.384082    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.437346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.437273    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hb7\" (UniqueName: \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\") pod \"maas-api-key-cleanup-29889960-48q7x\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.437346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.437293    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wsvc\" (UniqueName: \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\") pod \"maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.437346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.437282    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq446\" (UniqueName: \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\") pod \"maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.437531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.437386    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdm6\" (UniqueName: \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.438476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.438452    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgsp\" (UniqueName: \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.507609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.507581    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:00.515913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.515888    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:00.523576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.523554    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:00.551646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.551584    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:00.558826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.556594    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:00.942578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.942544    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:00.947749 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:00.947721    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bca4e9_dbb5_4963_8080_746dd553716f.slice/crio-e04ecb7a65e7ac4f34cc62084047685f974260ae376fde2ec9e5ca6955e1eb2e WatchSource:0}: Error finding container e04ecb7a65e7ac4f34cc62084047685f974260ae376fde2ec9e5ca6955e1eb2e: Status 404 returned error can't find the container with id e04ecb7a65e7ac4f34cc62084047685f974260ae376fde2ec9e5ca6955e1eb2e
Oct 30 22:00:00.958012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.957990    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:00.962371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:00.962333    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:00.965639 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:00.965619    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08105d36_d482_4cb1_9ecd_b12439f9024b.slice/crio-b428b0c6d160eaf06f8d4c79806b971f2949a0db3a4b88ca81ebc9433f81b34a WatchSource:0}: Error finding container b428b0c6d160eaf06f8d4c79806b971f2949a0db3a4b88ca81ebc9433f81b34a: Status 404 returned error can't find the container with id b428b0c6d160eaf06f8d4c79806b971f2949a0db3a4b88ca81ebc9433f81b34a
Oct 30 22:00:01.003104 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:01.003078    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f1d6eb_51d0_4fb0_9d23_6ed9cb2b31ba.slice/crio-c4838db11533390a2f2ac3560946d0d5e36d58b0c131d05e3d18f8ff4e5d9e54 WatchSource:0}: Error finding container c4838db11533390a2f2ac3560946d0d5e36d58b0c131d05e3d18f8ff4e5d9e54: Status 404 returned error can't find the container with id c4838db11533390a2f2ac3560946d0d5e36d58b0c131d05e3d18f8ff4e5d9e54
Oct 30 22:00:01.013850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.013819    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:01.019307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.019280    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:01.020917 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:01.020702    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d75824_8ed9_408d_96cb_cb8a5c514524.slice/crio-81a3281c1f627477e13bb3b0a2dfdadcc99a91ad0f6dd714e6d10aae1be08e5c WatchSource:0}: Error finding container 81a3281c1f627477e13bb3b0a2dfdadcc99a91ad0f6dd714e6d10aae1be08e5c: Status 404 returned error can't find the container with id 81a3281c1f627477e13bb3b0a2dfdadcc99a91ad0f6dd714e6d10aae1be08e5c
Oct 30 22:00:01.023907 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:01.023877    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ccab38_c8bd_40b1_b19a_a68cd9a6e669.slice/crio-4650ce4b2191d9832791e6efa8d4fecaf754cc749eef4bd37830c2063a5e7ace WatchSource:0}: Error finding container 4650ce4b2191d9832791e6efa8d4fecaf754cc749eef4bd37830c2063a5e7ace: Status 404 returned error can't find the container with id 4650ce4b2191d9832791e6efa8d4fecaf754cc749eef4bd37830c2063a5e7ace
Oct 30 22:00:01.619499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.619459    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerStarted","Data":"58c40d27058d996f7e53eb8e061b8ddaad05587739e88ff953e735044241decb"}
Oct 30 22:00:01.619930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.619523    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerStarted","Data":"81a3281c1f627477e13bb3b0a2dfdadcc99a91ad0f6dd714e6d10aae1be08e5c"}
Oct 30 22:00:01.621164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.621136    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerStarted","Data":"5a8b1c198b9e88447e324f199694cdd6e28d5ec6a20c4e41e9585fc8e4421174"}
Oct 30 22:00:01.621293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.621204    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerStarted","Data":"e04ecb7a65e7ac4f34cc62084047685f974260ae376fde2ec9e5ca6955e1eb2e"}
Oct 30 22:00:01.622817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.622770    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerStarted","Data":"f4827b8efd23d0ca45d2165ed9b08d8877193d4c575305b6ea21b1318c01bf74"}
Oct 30 22:00:01.622920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.622835    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerStarted","Data":"c4838db11533390a2f2ac3560946d0d5e36d58b0c131d05e3d18f8ff4e5d9e54"}
Oct 30 22:00:01.624290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.624268    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerStarted","Data":"2bf1e9742e47f97b1819c463825e9db17cf1f8952d0e68b2c6fe932892d27d26"}
Oct 30 22:00:01.624360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.624311    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerStarted","Data":"4650ce4b2191d9832791e6efa8d4fecaf754cc749eef4bd37830c2063a5e7ace"}
Oct 30 22:00:01.625912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.625890    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerStarted","Data":"7804d74c6c769bf27f3808f211161d2d7f6751269aae2294ba98e8123fcc61c2"}
Oct 30 22:00:01.626008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.625931    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerStarted","Data":"b428b0c6d160eaf06f8d4c79806b971f2949a0db3a4b88ca81ebc9433f81b34a"}
Oct 30 22:00:01.726239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.726189    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" podStartSLOduration=1.726174978 podStartE2EDuration="1.726174978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.725671092 +0000 UTC m=+1010.709577967" watchObservedRunningTime="2026-10-30 22:00:01.726174978 +0000 UTC m=+1010.710081855"
Oct 30 22:00:01.836881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.836756    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" podStartSLOduration=1.836741359 podStartE2EDuration="1.836741359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.833621374 +0000 UTC m=+1010.817528250" watchObservedRunningTime="2026-10-30 22:00:01.836741359 +0000 UTC m=+1010.820648235"
Oct 30 22:00:01.941545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:01.941495    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" podStartSLOduration=1.941480544 podStartE2EDuration="1.941480544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.939025964 +0000 UTC m=+1010.922932839" watchObservedRunningTime="2026-10-30 22:00:01.941480544 +0000 UTC m=+1010.925387443"
Oct 30 22:00:02.047008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.046954    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" podStartSLOduration=2.046940185 podStartE2EDuration="2.046940185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.046401694 +0000 UTC m=+1011.030308573" watchObservedRunningTime="2026-10-30 22:00:02.046940185 +0000 UTC m=+1011.030847061"
Oct 30 22:00:02.156534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.156482    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" podStartSLOduration=2.156468376 podStartE2EDuration="2.156468376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.152570304 +0000 UTC m=+1011.136477180" watchObservedRunningTime="2026-10-30 22:00:02.156468376 +0000 UTC m=+1011.140375252"
Oct 30 22:00:02.634987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.634949    5510 generic.go:356] "Generic (PLEG): container finished" podID="44bca4e9-dbb5-4963-8080-746dd553716f" containerID="5a8b1c198b9e88447e324f199694cdd6e28d5ec6a20c4e41e9585fc8e4421174" exitCode=60
Oct 30 22:00:02.635424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.634991    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerDied","Data":"5a8b1c198b9e88447e324f199694cdd6e28d5ec6a20c4e41e9585fc8e4421174"}
Oct 30 22:00:02.635424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.635292    5510 scope.go:122] "RemoveContainer" containerID="5a8b1c198b9e88447e324f199694cdd6e28d5ec6a20c4e41e9585fc8e4421174"
Oct 30 22:00:02.636987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.636966    5510 generic.go:356] "Generic (PLEG): container finished" podID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerID="f4827b8efd23d0ca45d2165ed9b08d8877193d4c575305b6ea21b1318c01bf74" exitCode=60
Oct 30 22:00:02.637089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.637058    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerDied","Data":"f4827b8efd23d0ca45d2165ed9b08d8877193d4c575305b6ea21b1318c01bf74"}
Oct 30 22:00:02.637270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.637251    5510 scope.go:122] "RemoveContainer" containerID="f4827b8efd23d0ca45d2165ed9b08d8877193d4c575305b6ea21b1318c01bf74"
Oct 30 22:00:02.638924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.638744    5510 generic.go:356] "Generic (PLEG): container finished" podID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerID="2bf1e9742e47f97b1819c463825e9db17cf1f8952d0e68b2c6fe932892d27d26" exitCode=60
Oct 30 22:00:02.638924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.638872    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerDied","Data":"2bf1e9742e47f97b1819c463825e9db17cf1f8952d0e68b2c6fe932892d27d26"}
Oct 30 22:00:02.639057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.639038    5510 scope.go:122] "RemoveContainer" containerID="2bf1e9742e47f97b1819c463825e9db17cf1f8952d0e68b2c6fe932892d27d26"
Oct 30 22:00:02.640552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.640535    5510 generic.go:356] "Generic (PLEG): container finished" podID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerID="7804d74c6c769bf27f3808f211161d2d7f6751269aae2294ba98e8123fcc61c2" exitCode=60
Oct 30 22:00:02.640630 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.640597    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerDied","Data":"7804d74c6c769bf27f3808f211161d2d7f6751269aae2294ba98e8123fcc61c2"}
Oct 30 22:00:02.640826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.640810    5510 scope.go:122] "RemoveContainer" containerID="7804d74c6c769bf27f3808f211161d2d7f6751269aae2294ba98e8123fcc61c2"
Oct 30 22:00:02.642287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.642266    5510 generic.go:356] "Generic (PLEG): container finished" podID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerID="58c40d27058d996f7e53eb8e061b8ddaad05587739e88ff953e735044241decb" exitCode=60
Oct 30 22:00:02.642385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.642296    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerDied","Data":"58c40d27058d996f7e53eb8e061b8ddaad05587739e88ff953e735044241decb"}
Oct 30 22:00:02.642510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.642498    5510 scope.go:122] "RemoveContainer" containerID="58c40d27058d996f7e53eb8e061b8ddaad05587739e88ff953e735044241decb"
Oct 30 22:00:02.763728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:02.763690    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:03.650959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.650927    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerStarted","Data":"ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"}
Oct 30 22:00:03.654476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.654448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerStarted","Data":"183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"}
Oct 30 22:00:03.656878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.656853    5510 generic.go:356] "Generic (PLEG): container finished" podID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerID="0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94" exitCode=60
Oct 30 22:00:03.656969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.656933    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerDied","Data":"0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94"}
Oct 30 22:00:03.657040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.656996    5510 scope.go:122] "RemoveContainer" containerID="58c40d27058d996f7e53eb8e061b8ddaad05587739e88ff953e735044241decb"
Oct 30 22:00:03.657271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.657188    5510 scope.go:122] "RemoveContainer" containerID="0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94"
Oct 30 22:00:03.657539 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:03.657513    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck_odh-ai-gateway-infra(30d75824-8ed9-408d-96cb-cb8a5c514524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524"
Oct 30 22:00:03.660493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.659828    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerStarted","Data":"4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"}
Oct 30 22:00:03.662493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.662468    5510 generic.go:356] "Generic (PLEG): container finished" podID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerID="758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab" exitCode=60
Oct 30 22:00:03.662603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.662563    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerDied","Data":"758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab"}
Oct 30 22:00:03.662752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.662736    5510 scope.go:122] "RemoveContainer" containerID="758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab"
Oct 30 22:00:03.662993 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:03.662974    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs_odh-ai-gateway-infra(d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba"
Oct 30 22:00:03.929920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:03.929895    5510 scope.go:122] "RemoveContainer" containerID="f4827b8efd23d0ca45d2165ed9b08d8877193d4c575305b6ea21b1318c01bf74"
Oct 30 22:00:04.671386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.671351    5510 generic.go:356] "Generic (PLEG): container finished" podID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerID="ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b" exitCode=60
Oct 30 22:00:04.671876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.671434    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerDied","Data":"ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"}
Oct 30 22:00:04.671876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.671503    5510 scope.go:122] "RemoveContainer" containerID="2bf1e9742e47f97b1819c463825e9db17cf1f8952d0e68b2c6fe932892d27d26"
Oct 30 22:00:04.671876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.671686    5510 scope.go:122] "RemoveContainer" containerID="ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"
Oct 30 22:00:04.672018 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:04.671955    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf_odh-ai-gateway-infra(d3ccab38-c8bd-40b1-b19a-a68cd9a6e669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669"
Oct 30 22:00:04.673562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.673543    5510 generic.go:356] "Generic (PLEG): container finished" podID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerID="183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294" exitCode=60
Oct 30 22:00:04.673657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.673591    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerDied","Data":"183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"}
Oct 30 22:00:04.673900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.673884    5510 scope.go:122] "RemoveContainer" containerID="183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"
Oct 30 22:00:04.674156 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:04.674139    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-48q7x_odh-ai-gateway-infra(08105d36-d482-4cb1-9ecd-b12439f9024b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b"
Oct 30 22:00:04.675352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.675336    5510 scope.go:122] "RemoveContainer" containerID="0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94"
Oct 30 22:00:04.675539 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:04.675522    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck_odh-ai-gateway-infra(30d75824-8ed9-408d-96cb-cb8a5c514524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524"
Oct 30 22:00:04.677027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.677008    5510 generic.go:356] "Generic (PLEG): container finished" podID="44bca4e9-dbb5-4963-8080-746dd553716f" containerID="4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024" exitCode=60
Oct 30 22:00:04.677112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.677091    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerDied","Data":"4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"}
Oct 30 22:00:04.677252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.677237    5510 scope.go:122] "RemoveContainer" containerID="4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"
Oct 30 22:00:04.677453 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:04.677429    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n_odh-ai-gateway-infra(44bca4e9-dbb5-4963-8080-746dd553716f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" podUID="44bca4e9-dbb5-4963-8080-746dd553716f"
Oct 30 22:00:04.678812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.678795    5510 scope.go:122] "RemoveContainer" containerID="758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab"
Oct 30 22:00:04.679025 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:04.679000    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs_odh-ai-gateway-infra(d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba"
Oct 30 22:00:04.687793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.687752    5510 scope.go:122] "RemoveContainer" containerID="7804d74c6c769bf27f3808f211161d2d7f6751269aae2294ba98e8123fcc61c2"
Oct 30 22:00:04.700503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:04.700478    5510 scope.go:122] "RemoveContainer" containerID="5a8b1c198b9e88447e324f199694cdd6e28d5ec6a20c4e41e9585fc8e4421174"
Oct 30 22:00:05.689334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:05.689302    5510 scope.go:122] "RemoveContainer" containerID="4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"
Oct 30 22:00:05.689831 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:05.689658    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n_odh-ai-gateway-infra(44bca4e9-dbb5-4963-8080-746dd553716f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" podUID="44bca4e9-dbb5-4963-8080-746dd553716f"
Oct 30 22:00:05.691231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:05.691198    5510 scope.go:122] "RemoveContainer" containerID="ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"
Oct 30 22:00:05.691472 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:05.691449    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf_odh-ai-gateway-infra(d3ccab38-c8bd-40b1-b19a-a68cd9a6e669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669"
Oct 30 22:00:05.692947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:05.692928    5510 scope.go:122] "RemoveContainer" containerID="183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"
Oct 30 22:00:05.693136 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:05.693119    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-48q7x_odh-ai-gateway-infra(08105d36-d482-4cb1-9ecd-b12439f9024b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b"
Oct 30 22:00:07.238642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:07.238609    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:09.132270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.132234    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:00:09.148493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.148459    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219424    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219462    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219492    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219717    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219765    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219901    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.219980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshpk\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.220038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.219987    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.220038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.220027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.246834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.246806    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:00:09.321082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321049    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321091    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321136    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshpk\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321315    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321387    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321435    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321482    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321526    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321545    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321525    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.321849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.321829    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.322027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.322006    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.323747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.323719    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.323979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.323958    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.377252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.377224    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.377523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.377501    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshpk\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\") pod \"e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") " pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.463357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.463333    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:09.891681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.891584    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:00:09.897498 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:09.897468    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3b11ae_eab3_4f2a_ad97_141b5f16e7ba.slice/crio-1f3d495d847271332d33b3371fae1b1993095e4ba164e98695d70e3e2d0f19a1 WatchSource:0}: Error finding container 1f3d495d847271332d33b3371fae1b1993095e4ba164e98695d70e3e2d0f19a1: Status 404 returned error can't find the container with id 1f3d495d847271332d33b3371fae1b1993095e4ba164e98695d70e3e2d0f19a1
Oct 30 22:00:09.900181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.900149    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:09.900321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.900218    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:09.900321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:09.900251    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:10.733200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:10.733160    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" event={"ID":"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba","Type":"ContainerStarted","Data":"5ec4029baa94f8b29c38d5b4768a62afb424ebf4b2925aead2f8fe6d36899294"}
Oct 30 22:00:10.733710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:10.733235    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" event={"ID":"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba","Type":"ContainerStarted","Data":"1f3d495d847271332d33b3371fae1b1993095e4ba164e98695d70e3e2d0f19a1"}
Oct 30 22:00:10.896401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:10.896350    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podStartSLOduration=1.8963365479999998 podStartE2EDuration="1.896336548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:10.893310513 +0000 UTC m=+1019.877217392" watchObservedRunningTime="2026-10-30 22:00:10.896336548 +0000 UTC m=+1019.880243485"
Oct 30 22:00:11.466804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:11.465102    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:11.470800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:11.468909    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:11.470800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:11.468975    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:11.742150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:11.742073    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:12.470804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:12.464198    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:12.470804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:12.464258    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:13.305429 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:13.305390    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:00:13.466831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:13.464220    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:13.466831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:13.464288    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:13.573001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:13.558229    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:14.468927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:14.468883    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:14.469535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:14.469500    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:15.471806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:15.468122    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:15.471806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:15.468182    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:16.118026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.117991    5510 scope.go:122] "RemoveContainer" containerID="183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"
Oct 30 22:00:16.118183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.118113    5510 scope.go:122] "RemoveContainer" containerID="758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab"
Oct 30 22:00:16.307797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.307746    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:16.464500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.464470    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:00:16.464670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.464519    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:00:16.812525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.812477    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerStarted","Data":"c10c6f2989d76d3be6199ff6689cbd0b3d2008ee020ac73bdd1dacbeb746f69a"}
Oct 30 22:00:16.814948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:16.814917    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerStarted","Data":"0592a5ea13588428f59104f54dfd763d0f7d19aeceec75d1b0f24a2d1e20c21f"}
Oct 30 22:00:17.127769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.127738    5510 scope.go:122] "RemoveContainer" containerID="0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94"
Oct 30 22:00:17.127997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.127867    5510 scope.go:122] "RemoveContainer" containerID="ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"
Oct 30 22:00:17.469509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.469482    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:17.469901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.469878    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:17.470596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.470578    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:00:17.823992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.823962    5510 generic.go:356] "Generic (PLEG): container finished" podID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerID="c10c6f2989d76d3be6199ff6689cbd0b3d2008ee020ac73bdd1dacbeb746f69a" exitCode=60
Oct 30 22:00:17.824421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.824035    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerDied","Data":"c10c6f2989d76d3be6199ff6689cbd0b3d2008ee020ac73bdd1dacbeb746f69a"}
Oct 30 22:00:17.824421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.824090    5510 scope.go:122] "RemoveContainer" containerID="758494a5755716194f4206dbdec7f150e3041d66a6ce9b134a9f32bfa3d988ab"
Oct 30 22:00:17.824562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.824433    5510 scope.go:122] "RemoveContainer" containerID="c10c6f2989d76d3be6199ff6689cbd0b3d2008ee020ac73bdd1dacbeb746f69a"
Oct 30 22:00:17.824771 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:17.824749    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs_odh-ai-gateway-infra(d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba"
Oct 30 22:00:17.826127 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.826105    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerStarted","Data":"b779176c48ce99909bab99c7c05131e6be3735f1f1a2d208e764755be6597a2e"}
Oct 30 22:00:17.827914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.827894    5510 generic.go:356] "Generic (PLEG): container finished" podID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerID="0592a5ea13588428f59104f54dfd763d0f7d19aeceec75d1b0f24a2d1e20c21f" exitCode=60
Oct 30 22:00:17.828012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.827969    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerDied","Data":"0592a5ea13588428f59104f54dfd763d0f7d19aeceec75d1b0f24a2d1e20c21f"}
Oct 30 22:00:17.828325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.828308    5510 scope.go:122] "RemoveContainer" containerID="0592a5ea13588428f59104f54dfd763d0f7d19aeceec75d1b0f24a2d1e20c21f"
Oct 30 22:00:17.828550 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:17.828532    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-48q7x_odh-ai-gateway-infra(08105d36-d482-4cb1-9ecd-b12439f9024b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b"
Oct 30 22:00:17.830146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.830123    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerStarted","Data":"d3e3339e5a11464968a6737caff26094d6609adf142715bbb6510bbfed432dd6"}
Oct 30 22:00:17.837666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.837650    5510 scope.go:122] "RemoveContainer" containerID="183b1cf739fd7dd18408a17ae4dd96b67c8f6d7b0bf96c0987a4ba0251c82294"
Oct 30 22:00:17.931654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:17.931621    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:18.039550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.039516    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:18.118345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.118275    5510 scope.go:122] "RemoveContainer" containerID="4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"
Oct 30 22:00:18.839148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.839064    5510 generic.go:356] "Generic (PLEG): container finished" podID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerID="d3e3339e5a11464968a6737caff26094d6609adf142715bbb6510bbfed432dd6" exitCode=60
Oct 30 22:00:18.839566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.839138    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerDied","Data":"d3e3339e5a11464968a6737caff26094d6609adf142715bbb6510bbfed432dd6"}
Oct 30 22:00:18.839566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.839234    5510 scope.go:122] "RemoveContainer" containerID="0bb452f0e44da17259b584cc6528280b64d49a7abd23665af7c04d7fb8070c94"
Oct 30 22:00:18.839718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.839597    5510 scope.go:122] "RemoveContainer" containerID="d3e3339e5a11464968a6737caff26094d6609adf142715bbb6510bbfed432dd6"
Oct 30 22:00:18.839964 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:18.839940    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck_odh-ai-gateway-infra(30d75824-8ed9-408d-96cb-cb8a5c514524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524"
Oct 30 22:00:18.841263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.841237    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerStarted","Data":"038bd10034db4ddc12a3754b4cba26317f9b04d4b82a60fe09a678d3af1bfb47"}
Oct 30 22:00:18.844825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.844804    5510 generic.go:356] "Generic (PLEG): container finished" podID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerID="b779176c48ce99909bab99c7c05131e6be3735f1f1a2d208e764755be6597a2e" exitCode=60
Oct 30 22:00:18.844912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.844874    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerDied","Data":"b779176c48ce99909bab99c7c05131e6be3735f1f1a2d208e764755be6597a2e"}
Oct 30 22:00:18.845252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.845234    5510 scope.go:122] "RemoveContainer" containerID="b779176c48ce99909bab99c7c05131e6be3735f1f1a2d208e764755be6597a2e"
Oct 30 22:00:18.845564 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:18.845538    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf_odh-ai-gateway-infra(d3ccab38-c8bd-40b1-b19a-a68cd9a6e669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669"
Oct 30 22:00:18.854572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:18.854537    5510 scope.go:122] "RemoveContainer" containerID="ec7bda29a1726f496d9e39f8e65997d2887bebea056e6ae1d5b7194cbe4c0f3b"
Oct 30 22:00:19.002826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.002620    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:19.073097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.073069    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:19.137736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.137668    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\") pod \"08105d36-d482-4cb1-9ecd-b12439f9024b\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") "
Oct 30 22:00:19.137736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.137718    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\") pod \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") "
Oct 30 22:00:19.137932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.137906    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\" (UniqueName: \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\") pod \"08105d36-d482-4cb1-9ecd-b12439f9024b\" (UID: \"08105d36-d482-4cb1-9ecd-b12439f9024b\") "
Oct 30 22:00:19.137987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.137947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca" pod "08105d36-d482-4cb1-9ecd-b12439f9024b" (UID: "08105d36-d482-4cb1-9ecd-b12439f9024b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.138041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.138023    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\" (UniqueName: \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\") pod \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\" (UID: \"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba\") "
Oct 30 22:00:19.138101 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.138050    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca" pod "d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" (UID: "d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.138425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.138407    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08105d36-d482-4cb1-9ecd-b12439f9024b-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.138506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.138427    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.139985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.139966    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7" pod "08105d36-d482-4cb1-9ecd-b12439f9024b" (UID: "08105d36-d482-4cb1-9ecd-b12439f9024b"). InnerVolumeSpecName "kube-api-access-99hb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.140048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.140006    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6" pod "d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" (UID: "d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba"). InnerVolumeSpecName "kube-api-access-kxdm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.179286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.179262    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:19.239263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.239221    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hb7\" (UniqueName: \"kubernetes.io/projected/08105d36-d482-4cb1-9ecd-b12439f9024b-kube-api-access-99hb7\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.239263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.239260    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdm6\" (UniqueName: \"kubernetes.io/projected/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba-kube-api-access-kxdm6\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.301852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.301792    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:19.856624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.856595    5510 generic.go:356] "Generic (PLEG): container finished" podID="44bca4e9-dbb5-4963-8080-746dd553716f" containerID="038bd10034db4ddc12a3754b4cba26317f9b04d4b82a60fe09a678d3af1bfb47" exitCode=60
Oct 30 22:00:19.857032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.856681    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerDied","Data":"038bd10034db4ddc12a3754b4cba26317f9b04d4b82a60fe09a678d3af1bfb47"}
Oct 30 22:00:19.857032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.856774    5510 scope.go:122] "RemoveContainer" containerID="4f3b8ad8b37ed93d74b286da38422d96085af60e35c986841c27ba455f968024"
Oct 30 22:00:19.857188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.857172    5510 scope.go:122] "RemoveContainer" containerID="038bd10034db4ddc12a3754b4cba26317f9b04d4b82a60fe09a678d3af1bfb47"
Oct 30 22:00:19.857471 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:19.857451    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n_odh-ai-gateway-infra(44bca4e9-dbb5-4963-8080-746dd553716f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" podUID="44bca4e9-dbb5-4963-8080-746dd553716f"
Oct 30 22:00:19.859085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.859060    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"
Oct 30 22:00:19.859232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.859103    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs" event={"ID":"d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba","Type":"ContainerDied","Data":"c4838db11533390a2f2ac3560946d0d5e36d58b0c131d05e3d18f8ff4e5d9e54"}
Oct 30 22:00:19.871964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.871943    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"
Oct 30 22:00:19.872457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.871934    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x" event={"ID":"08105d36-d482-4cb1-9ecd-b12439f9024b","Type":"ContainerDied","Data":"b428b0c6d160eaf06f8d4c79806b971f2949a0db3a4b88ca81ebc9433f81b34a"}
Oct 30 22:00:19.873590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.873571    5510 scope.go:122] "RemoveContainer" containerID="c10c6f2989d76d3be6199ff6689cbd0b3d2008ee020ac73bdd1dacbeb746f69a"
Oct 30 22:00:19.891882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:19.891838    5510 scope.go:122] "RemoveContainer" containerID="0592a5ea13588428f59104f54dfd763d0f7d19aeceec75d1b0f24a2d1e20c21f"
Oct 30 22:00:20.038794 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.038755    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:20.074327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.074298    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:20.082047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.082019    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-48q7x"]
Oct 30 22:00:20.109681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.109635    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:20.150381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150357    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\") pod \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") "
Oct 30 22:00:20.150483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\" (UniqueName: \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\") pod \"30d75824-8ed9-408d-96cb-cb8a5c514524\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") "
Oct 30 22:00:20.150531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\") pod \"30d75824-8ed9-408d-96cb-cb8a5c514524\" (UID: \"30d75824-8ed9-408d-96cb-cb8a5c514524\") "
Oct 30 22:00:20.150590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150578    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\" (UniqueName: \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\") pod \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\" (UID: \"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669\") "
Oct 30 22:00:20.150834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150713    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca" pod "d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" (UID: "d3ccab38-c8bd-40b1-b19a-a68cd9a6e669"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.150907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150878    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca" pod "30d75824-8ed9-408d-96cb-cb8a5c514524" (UID: "30d75824-8ed9-408d-96cb-cb8a5c514524"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.150955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.150927    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.152588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.152567    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp" pod "30d75824-8ed9-408d-96cb-cb8a5c514524" (UID: "30d75824-8ed9-408d-96cb-cb8a5c514524"). InnerVolumeSpecName "kube-api-access-4cgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.152758 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.152737    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc" pod "d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" (UID: "d3ccab38-c8bd-40b1-b19a-a68cd9a6e669"). InnerVolumeSpecName "kube-api-access-9wsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.174825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.174800    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:20.251840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.251773    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d75824-8ed9-408d-96cb-cb8a5c514524-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.251840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.251845    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wsvc\" (UniqueName: \"kubernetes.io/projected/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669-kube-api-access-9wsvc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.252086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.251862    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cgsp\" (UniqueName: \"kubernetes.io/projected/30d75824-8ed9-408d-96cb-cb8a5c514524-kube-api-access-4cgsp\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.397901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.397808    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:20.412105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.412076    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889960-d4txs"]
Oct 30 22:00:20.812871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.812836    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:20.936244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.936202    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf" event={"ID":"d3ccab38-c8bd-40b1-b19a-a68cd9a6e669","Type":"ContainerDied","Data":"4650ce4b2191d9832791e6efa8d4fecaf754cc749eef4bd37830c2063a5e7ace"}
Oct 30 22:00:20.936866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.936841    5510 scope.go:122] "RemoveContainer" containerID="b779176c48ce99909bab99c7c05131e6be3735f1f1a2d208e764755be6597a2e"
Oct 30 22:00:20.937178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.937156    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"
Oct 30 22:00:20.942217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.942189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck" event={"ID":"30d75824-8ed9-408d-96cb-cb8a5c514524","Type":"ContainerDied","Data":"81a3281c1f627477e13bb3b0a2dfdadcc99a91ad0f6dd714e6d10aae1be08e5c"}
Oct 30 22:00:20.942314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.942285    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"
Oct 30 22:00:20.951464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:20.951442    5510 scope.go:122] "RemoveContainer" containerID="d3e3339e5a11464968a6737caff26094d6609adf142715bbb6510bbfed432dd6"
Oct 30 22:00:21.098845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.098816    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:21.103676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.103652    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889960-58rck"]
Oct 30 22:00:21.106497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.106481    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:21.125430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.125402    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" path="/var/lib/kubelet/pods/08105d36-d482-4cb1-9ecd-b12439f9024b/volumes"
Oct 30 22:00:21.125889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.125874    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" path="/var/lib/kubelet/pods/30d75824-8ed9-408d-96cb-cb8a5c514524/volumes"
Oct 30 22:00:21.126263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.126249    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" path="/var/lib/kubelet/pods/d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba/volumes"
Oct 30 22:00:21.172326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.172293    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\" (UniqueName: \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\") pod \"44bca4e9-dbb5-4963-8080-746dd553716f\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") "
Oct 30 22:00:21.172462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.172404    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\") pod \"44bca4e9-dbb5-4963-8080-746dd553716f\" (UID: \"44bca4e9-dbb5-4963-8080-746dd553716f\") "
Oct 30 22:00:21.172856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.172824    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca" pod "44bca4e9-dbb5-4963-8080-746dd553716f" (UID: "44bca4e9-dbb5-4963-8080-746dd553716f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.172992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.172934    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44bca4e9-dbb5-4963-8080-746dd553716f-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.174480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.174456    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446" pod "44bca4e9-dbb5-4963-8080-746dd553716f" (UID: "44bca4e9-dbb5-4963-8080-746dd553716f"). InnerVolumeSpecName "kube-api-access-vq446". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.273675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.273641    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq446\" (UniqueName: \"kubernetes.io/projected/44bca4e9-dbb5-4963-8080-746dd553716f-kube-api-access-vq446\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.304735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.304703    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:21.311496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.311465    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5402dd-29889960-hn7zf"]
Oct 30 22:00:21.957350 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.957312    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n" event={"ID":"44bca4e9-dbb5-4963-8080-746dd553716f","Type":"ContainerDied","Data":"e04ecb7a65e7ac4f34cc62084047685f974260ae376fde2ec9e5ca6955e1eb2e"}
Oct 30 22:00:21.957889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.957387    5510 scope.go:122] "RemoveContainer" containerID="038bd10034db4ddc12a3754b4cba26317f9b04d4b82a60fe09a678d3af1bfb47"
Oct 30 22:00:21.957889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:21.957323    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"
Oct 30 22:00:22.113074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:22.113036    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:22.118363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:22.118335    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a07620-29889960-q4p7n"]
Oct 30 22:00:22.510439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:22.510406    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:23.126496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:23.126460    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" path="/var/lib/kubelet/pods/44bca4e9-dbb5-4963-8080-746dd553716f/volumes"
Oct 30 22:00:23.127056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:23.127039    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" path="/var/lib/kubelet/pods/d3ccab38-c8bd-40b1-b19a-a68cd9a6e669/volumes"
Oct 30 22:00:23.879731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:23.879697    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:00:25.313589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:25.313561    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:27.515147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515110    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:00:27.515900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515879    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515901    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515928    5510 cpu_manager.go:414] "containerMap: removing container" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515935    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515948    5510 cpu_manager.go:414] "containerMap: removing container" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515956    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515965    5510 cpu_manager.go:414] "containerMap: removing container" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515973    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515986    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.515993    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516002    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516009    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516027    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516035    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516052    5510 cpu_manager.go:414] "containerMap: removing container" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516059    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516074    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516081    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516096    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516103    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516125    5510 cpu_manager.go:414] "containerMap: removing container" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516132    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516266    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516281    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516290    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516300    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516310    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516320    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516327    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516336    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516346    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516354    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516364    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:27.516681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.516373    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:27.520306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.520285    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.579096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.577514    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:00:27.617421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.617384    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f776cb\"/\"distinct-w3-f776cb-kserve-self-signed-certs\""
Oct 30 22:00:27.635486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.635452    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:27.681421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.676368    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.694964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.693858    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5q8\" (UniqueName: \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.694964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.693946    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.694964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.694044    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.694964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.694121    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.694964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.694220    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.798252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.798164    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.798486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.798466    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5q8\" (UniqueName: \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.798620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.798604    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.798759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.798745    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.798907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.798893    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.799054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.799042    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.799583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.799561    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.800359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.800337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.800870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.800821    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.803629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.803600    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.805186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.805165    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:27.852385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:27.852357    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5q8\" (UniqueName: \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\") pod \"distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") " pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:28.133248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:28.133167    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:28.513114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:28.513083    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:00:28.518563 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:28.518535    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1262f04_8f4e_4c25_87ca_99dc7013c6db.slice/crio-6808d7942a9b9f02d1ca01ef379270593a60a39b5227bbc036d00ccf3c027ba3 WatchSource:0}: Error finding container 6808d7942a9b9f02d1ca01ef379270593a60a39b5227bbc036d00ccf3c027ba3: Status 404 returned error can't find the container with id 6808d7942a9b9f02d1ca01ef379270593a60a39b5227bbc036d00ccf3c027ba3
Oct 30 22:00:29.017364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:29.017330    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" event={"ID":"d1262f04-8f4e-4c25-87ca-99dc7013c6db","Type":"ContainerStarted","Data":"a884b1b826da7d7b6bac49cfdbddd4b240869ac06a4f7cea4f4fbb462b77a450"}
Oct 30 22:00:29.017557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:29.017375    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" event={"ID":"d1262f04-8f4e-4c25-87ca-99dc7013c6db","Type":"ContainerStarted","Data":"6808d7942a9b9f02d1ca01ef379270593a60a39b5227bbc036d00ccf3c027ba3"}
Oct 30 22:00:29.128768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:29.128710    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podStartSLOduration=2.128690148 podStartE2EDuration="2.128690148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:29.123926098 +0000 UTC m=+1038.107832976" watchObservedRunningTime="2026-10-30 22:00:29.128690148 +0000 UTC m=+1038.112597025"
Oct 30 22:00:29.133644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:29.133620    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:29.725090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:29.725057    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:34.155370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:34.155328    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:38.133861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.133828    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:38.146496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.146464    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:38.602918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.602886    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:00:38.603459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603446    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:38.603510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603460    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:38.603510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603487    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:38.603510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603493    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:38.603510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603508    5510 cpu_manager.go:414] "containerMap: removing container" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:38.603624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603515    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08105d36-d482-4cb1-9ecd-b12439f9024b" containerName="cleanup"
Oct 30 22:00:38.603624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603525    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:38.603624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603529    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bca4e9-dbb5-4963-8080-746dd553716f" containerName="cleanup"
Oct 30 22:00:38.603624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603621    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f1d6eb-51d0-4fb0-9d23-6ed9cb2b31ba" containerName="cleanup"
Oct 30 22:00:38.603772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603631    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ccab38-c8bd-40b1-b19a-a68cd9a6e669" containerName="cleanup"
Oct 30 22:00:38.603772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.603637    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d75824-8ed9-408d-96cb-cb8a5c514524" containerName="cleanup"
Oct 30 22:00:38.607988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.607968    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.620769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.620745    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:38.707827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.707758    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:00:38.751892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.751860    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.751984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752047    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752096    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752138    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752162    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6p5\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752289    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752334    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.752428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.752410    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.853964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.853857    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.853962    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.853986    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6p5\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854010    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854031    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854095    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854190    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854290    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854337    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854341    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854400    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854614    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.854704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854665    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.855004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.854982    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.856388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.856361    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.856895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.856876    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.908263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.908235    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.908452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.908429    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6p5\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:38.922283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:38.922255    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:39.103125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:39.103093    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:00:39.387868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:39.387836    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:00:39.394312 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:00:39.394282    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20563428_9ca2_4c60_ab11_fb4930f95a61.slice/crio-400e860450585927a37ac998cb60d5ae988d8f80588c2af049b2cb54855cd157 WatchSource:0}: Error finding container 400e860450585927a37ac998cb60d5ae988d8f80588c2af049b2cb54855cd157: Status 404 returned error can't find the container with id 400e860450585927a37ac998cb60d5ae988d8f80588c2af049b2cb54855cd157
Oct 30 22:00:39.396838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:39.396805    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:39.396919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:39.396875    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:39.396919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:39.396905    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:00:40.101260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.101222    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" event={"ID":"20563428-9ca2-4c60-ab11-fb4930f95a61","Type":"ContainerStarted","Data":"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"}
Oct 30 22:00:40.101560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.101541    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" event={"ID":"20563428-9ca2-4c60-ab11-fb4930f95a61","Type":"ContainerStarted","Data":"400e860450585927a37ac998cb60d5ae988d8f80588c2af049b2cb54855cd157"}
Oct 30 22:00:40.216560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.216498    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podStartSLOduration=2.216479476 podStartE2EDuration="2.216479476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:40.213857966 +0000 UTC m=+1049.197764847" watchObservedRunningTime="2026-10-30 22:00:40.216479476 +0000 UTC m=+1049.200386353"
Oct 30 22:00:40.923024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.922982    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:40.924700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.924674    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:40.924852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:40.924733    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:41.479396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:41.479221    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:41.923031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:41.922944    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:41.923315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:41.923281    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:42.923443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:42.923411    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:42.923856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:42.923475    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:43.127455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:43.127423    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:43.313413 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:00:43.313369    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:00:43.922869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:43.922836    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:43.923052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:43.922891    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:44.922693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:44.922658    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:44.923088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:44.922723    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:45.923620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:45.923585    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:00:45.924006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:45.923641    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 22:00:46.927496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:46.927465    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:46.927987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:46.927875    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:46.928577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:46.928559    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:00:47.637882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:47.637846    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:51.527889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:51.527859    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:52.030797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:52.030751    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:56.517421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:00:56.517383    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:01.007910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:01.007867    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:03.002716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:03.002639    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:01:05.463361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:05.450476    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:09.931996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:09.931945    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:10.200216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:10.200131    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.284291 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:01:13.284246    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:01:14.426502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:14.426468    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:18.931114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:18.931079    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:22.053944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:22.053861    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"]
Oct 30 22:01:22.100280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:22.100248    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c9d442-tmsxc"]
Oct 30 22:01:22.859502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:22.859464    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.127498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.127413    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d" path="/var/lib/kubelet/pods/c21bc0f8-7b0c-43c3-9095-a95cfe7f1c8d/volumes"
Oct 30 22:01:23.261768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.261733    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:01:23.268357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.268316    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.312031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.311989    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.312259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.312234    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.312476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.312458    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.312697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.312678    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.312872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.312850    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.313038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.313020    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9mb\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.313194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.313177    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.313318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.313302    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.313461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.313442    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.416764    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.416938    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.416972    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417034    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9mb\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417105    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417157    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417201    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.417844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.417248    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.421812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.418945    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.421812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.419581    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.421812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.419847    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.421812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.420623    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.421812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.420912    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.437854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.423860    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.440115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.440046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:23.440793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.440260    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.550621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.513815    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:01:23.550621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.541196    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9mb\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.595799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.594363    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:23.598635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:23.598385    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:24.066754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.066721    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:01:24.072734 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:01:24.072703    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b5f0a4_9839_4721_841c_0cc6f7139c54.slice/crio-df397a49f3707e6cb0f3bd376fea531e8e62a702ed1f14af88fd78f3ebe0093c WatchSource:0}: Error finding container df397a49f3707e6cb0f3bd376fea531e8e62a702ed1f14af88fd78f3ebe0093c: Status 404 returned error can't find the container with id df397a49f3707e6cb0f3bd376fea531e8e62a702ed1f14af88fd78f3ebe0093c
Oct 30 22:01:24.075134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.075101    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:01:24.075245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.075182    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:01:24.075245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.075227    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:01:24.468616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.468572    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" event={"ID":"10b5f0a4-9839-4721-841c-0cc6f7139c54","Type":"ContainerStarted","Data":"bfbeda8cdc5826c9c3d15bdb0b185d149698b65a9d728f3d37d5238297bfc67f"}
Oct 30 22:01:24.469094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.468678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" event={"ID":"10b5f0a4-9839-4721-841c-0cc6f7139c54","Type":"ContainerStarted","Data":"df397a49f3707e6cb0f3bd376fea531e8e62a702ed1f14af88fd78f3ebe0093c"}
Oct 30 22:01:24.599327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.599290    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:24.629154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:24.629090    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podStartSLOduration=1.629076354 podStartE2EDuration="1.629076354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:24.62504709 +0000 UTC m=+1093.608953966" watchObservedRunningTime="2026-10-30 22:01:24.629076354 +0000 UTC m=+1093.612983229"
Oct 30 22:01:25.600959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:25.600927    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:25.601347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:25.600984    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:26.599768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:26.599734    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:26.599990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:26.599806    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:27.599250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:27.599215    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:27.599745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:27.599269    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:28.038679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:28.038642    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:28.599333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:28.599295    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:28.599740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:28.599361    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:29.599771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:29.599731    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:29.600181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:29.599825    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:30.125489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.125457    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"]
Oct 30 22:01:30.129929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.129906    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.186813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.186741    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"]
Oct 30 22:01:30.226063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.226027    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:30.226243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.226099    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:01:30.226243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.226108    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:01:30.226520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.226496    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:01:30.268875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.268832    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcnj\" (UniqueName: \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.269059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.269043    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.269189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.269173    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.371291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.371247    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.371462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.371395    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.371547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.371518    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcnj\" (UniqueName: \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.372263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.372223    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.374071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.374044    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.425381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.425317    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcnj\" (UniqueName: \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\") pod \"maas-api-revoke-sub-62fa0cd0-lzkts\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.444375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.444344    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:30.599829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.599792    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 22:01:30.600230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.599846    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 22:01:30.829437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:30.829404    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"]
Oct 30 22:01:30.835525 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:01:30.835487    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6619e8ef_27af_4283_9b7d_5686714b55e0.slice/crio-b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1 WatchSource:0}: Error finding container b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1: Status 404 returned error can't find the container with id b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1
Oct 30 22:01:31.550450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.550365    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts" event={"ID":"6619e8ef-27af-4283-9b7d-5686714b55e0","Type":"ContainerStarted","Data":"e69ab3569559166c48fba6b9efa6e0546ae17ba01927a97fc5d0b4f0a412d908"}
Oct 30 22:01:31.550450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.550445    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts" event={"ID":"6619e8ef-27af-4283-9b7d-5686714b55e0","Type":"ContainerStarted","Data":"b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1"}
Oct 30 22:01:31.603172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.603145    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:31.603577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.603410    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:31.604126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.604109    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:01:31.663374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:31.663327    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts" podStartSLOduration=1.663315126 podStartE2EDuration="1.663315126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:31.661017919 +0000 UTC m=+1100.644924796" watchObservedRunningTime="2026-10-30 22:01:31.663315126 +0000 UTC m=+1100.647222001"
Oct 30 22:01:32.525236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:32.525205    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:32.575698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:32.575657    5510 generic.go:356] "Generic (PLEG): container finished" podID="6619e8ef-27af-4283-9b7d-5686714b55e0" containerID="e69ab3569559166c48fba6b9efa6e0546ae17ba01927a97fc5d0b4f0a412d908" exitCode=0
Oct 30 22:01:32.577023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:32.576984    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts" event={"ID":"6619e8ef-27af-4283-9b7d-5686714b55e0","Type":"ContainerDied","Data":"e69ab3569559166c48fba6b9efa6e0546ae17ba01927a97fc5d0b4f0a412d908"}
Oct 30 22:01:33.732171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.732141    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:33.827587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.827496    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\") pod \"6619e8ef-27af-4283-9b7d-5686714b55e0\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") "
Oct 30 22:01:33.827754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.827593    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\" (UniqueName: \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\") pod \"6619e8ef-27af-4283-9b7d-5686714b55e0\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") "
Oct 30 22:01:33.827754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.827622    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\") pod \"6619e8ef-27af-4283-9b7d-5686714b55e0\" (UID: \"6619e8ef-27af-4283-9b7d-5686714b55e0\") "
Oct 30 22:01:33.828058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.828034    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca" pod "6619e8ef-27af-4283-9b7d-5686714b55e0" (UID: "6619e8ef-27af-4283-9b7d-5686714b55e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:33.828263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.828238    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:33.829576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.829549    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca" pod "6619e8ef-27af-4283-9b7d-5686714b55e0" (UID: "6619e8ef-27af-4283-9b7d-5686714b55e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:33.829680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.829666    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj" pod "6619e8ef-27af-4283-9b7d-5686714b55e0" (UID: "6619e8ef-27af-4283-9b7d-5686714b55e0"). InnerVolumeSpecName "kube-api-access-8jcnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:33.929627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.929593    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6619e8ef-27af-4283-9b7d-5686714b55e0-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:33.929627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:33.929622    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcnj\" (UniqueName: \"kubernetes.io/projected/6619e8ef-27af-4283-9b7d-5686714b55e0-kube-api-access-8jcnj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:34.592826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:34.592795    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"
Oct 30 22:01:34.592826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:34.592819    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts" event={"ID":"6619e8ef-27af-4283-9b7d-5686714b55e0","Type":"ContainerDied","Data":"b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1"}
Oct 30 22:01:34.593046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:34.592868    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00ba3706f5720267b50b5adc84ac2371b1f7867aeb377066c8d7524a97a12c1"
Oct 30 22:01:36.929595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:36.929559    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.429358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:41.429325    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:43.279879 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:01:43.279846    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:01:46.039399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:46.039356    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:46.236242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:46.236204    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:49.817528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:49.817495    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:01:50.518579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:50.518538    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:53.471360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:53.471317    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" podUID="0e90b7cbbf3b5bb50f8f3af03ed08b90" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 22:01:54.738117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.738083    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:01:54.738997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.738975    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6619e8ef-27af-4283-9b7d-5686714b55e0" containerName="revoke-keys"
Oct 30 22:01:54.738997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.738997    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6619e8ef-27af-4283-9b7d-5686714b55e0" containerName="revoke-keys"
Oct 30 22:01:54.739211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.739197    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6619e8ef-27af-4283-9b7d-5686714b55e0" containerName="revoke-keys"
Oct 30 22:01:54.743213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.743192    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.800859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.800823    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:01:54.820791 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.820753    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.820958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.820874    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.820958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.820944    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.821060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.820977    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.821060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.821017    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.821163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.821072    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjp9\" (UniqueName: \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.836311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.836279    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"openshift-service-ca.crt\""
Oct 30 22:01:54.836482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.836331    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"distinct-2-w4-17d809-kserve-self-signed-certs\""
Oct 30 22:01:54.836482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.836358    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"default-dockercfg-v7d58\""
Oct 30 22:01:54.840122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.840101    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-17d809\"/\"kube-root-ca.crt\""
Oct 30 22:01:54.860711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.860685    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:54.932032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.931992    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.944615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.944583    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.945038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.944999    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.945251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.945221    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.945456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.945439    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.950920    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjp9\" (UniqueName: \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.946446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.946693    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.949500    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.950062    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:54.951618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:54.946215    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:55.016135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.012950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjp9\" (UniqueName: \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\") pod \"distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") " pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:55.055447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.055417    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:55.101194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.101110    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:55.443266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.443185    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:01:55.448506 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:01:55.448478    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod344fc2e6_dda8_4078_a78e_1e9c66247915.slice/crio-59651daad7d0f6fa87852c25b273eee6b6c9fd182fdeb68b65a01c50b9d92575 WatchSource:0}: Error finding container 59651daad7d0f6fa87852c25b273eee6b6c9fd182fdeb68b65a01c50b9d92575: Status 404 returned error can't find the container with id 59651daad7d0f6fa87852c25b273eee6b6c9fd182fdeb68b65a01c50b9d92575
Oct 30 22:01:55.750157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.750122    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" event={"ID":"344fc2e6-dda8-4078-a78e-1e9c66247915","Type":"ContainerStarted","Data":"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"}
Oct 30 22:01:55.750157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.750165    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" event={"ID":"344fc2e6-dda8-4078-a78e-1e9c66247915","Type":"ContainerStarted","Data":"59651daad7d0f6fa87852c25b273eee6b6c9fd182fdeb68b65a01c50b9d92575"}
Oct 30 22:01:55.858449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:55.858383    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podStartSLOduration=1.858365206 podStartE2EDuration="1.858365206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:55.857772449 +0000 UTC m=+1124.841679328" watchObservedRunningTime="2026-10-30 22:01:55.858365206 +0000 UTC m=+1124.842272130"
Oct 30 22:01:56.057680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:56.055934    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:01:57.755263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.755225    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:01:57.760306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.760284    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.813229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.813198    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:01:57.854090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.854058    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5402dd\"/\"distinct-2-w5-5402dd-kserve-self-signed-certs\""
Oct 30 22:01:57.860821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.860797    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:57.925450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.925407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.925699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.925676    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclxw\" (UniqueName: \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.936941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.936905    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.937420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.937395    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.937934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.937914    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:57.938417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:57.938397    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.051713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051632    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.051713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051680    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclxw\" (UniqueName: \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.051713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051710    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051749    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051765    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.051928    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.052205    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.052297    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.052505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.052480    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.054203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.054176    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.055516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.055495    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.105925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.105899    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclxw\" (UniqueName: \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\") pod \"distinct-2-w5-5402dd-kserve-77566b86dd-55xd8\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") " pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.374558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.374476    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:01:58.760737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:58.760697    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:01:58.766664 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:01:58.766629    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4864ae_619c_46c3_a3f5_ca5143683565.slice/crio-35082210807b81670e9d48e749a41de1a3778e08d95ed656c01a0c4269e05402 WatchSource:0}: Error finding container 35082210807b81670e9d48e749a41de1a3778e08d95ed656c01a0c4269e05402: Status 404 returned error can't find the container with id 35082210807b81670e9d48e749a41de1a3778e08d95ed656c01a0c4269e05402
Oct 30 22:01:59.618509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:59.618476    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:59.780721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:59.780685    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" event={"ID":"aa4864ae-619c-46c3-a3f5-ca5143683565","Type":"ContainerStarted","Data":"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"}
Oct 30 22:01:59.781161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:59.780733    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" event={"ID":"aa4864ae-619c-46c3-a3f5-ca5143683565","Type":"ContainerStarted","Data":"35082210807b81670e9d48e749a41de1a3778e08d95ed656c01a0c4269e05402"}
Oct 30 22:01:59.890911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:01:59.890814    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podStartSLOduration=2.890770206 podStartE2EDuration="2.890770206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:59.887881881 +0000 UTC m=+1128.871788757" watchObservedRunningTime="2026-10-30 22:01:59.890770206 +0000 UTC m=+1128.874677139"
Oct 30 22:02:00.374869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:00.374837    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:01.045408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:01.045373    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"]
Oct 30 22:02:01.091430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:01.091396    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59c879e6-mph9h"]
Oct 30 22:02:01.125479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:01.125449    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62a2a50-94c4-4973-a64a-bc738e4c3865" path="/var/lib/kubelet/pods/e62a2a50-94c4-4973-a64a-bc738e4c3865/volumes"
Oct 30 22:02:03.989219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:03.989182    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:05.056499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:05.056463    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:02:05.073836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:05.073805    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:02:05.839863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:05.839823    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:02:08.374853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:08.374818    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:08.387765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:08.387737    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:08.440887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:08.440845    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:08.856426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:08.856391    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:10.046855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:10.046820    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"]
Oct 30 22:02:10.095051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:10.095017    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49606ce5-22sp6"]
Oct 30 22:02:11.126644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:11.126610    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5" path="/var/lib/kubelet/pods/0d9fb28c-9e7c-4909-a0c5-bdc61c4153b5/volumes"
Oct 30 22:02:12.922894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:12.922863    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:13.280977 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:02:13.280938    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:02:13.619870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:13.619723    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:14.961487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:14.961457    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"]
Oct 30 22:02:14.967333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:14.967314    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.019270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.019238    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"]
Oct 30 22:02:15.034472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.034446    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.034616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.034513    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.034752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.034718    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvwc\" (UniqueName: \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.064834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.060814    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:02:15.064834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.062775    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:15.064834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.062809    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:15.064834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.062832    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:15.136405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.136367    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.136594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.136457    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.136594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.136565    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvwc\" (UniqueName: \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.137328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.137301    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.139151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.139125    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.189920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.189886    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvwc\" (UniqueName: \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\") pod \"maas-api-revoke-sub-3588cb87-bdb7x\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.280446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.280362    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:15.767670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.767543    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"]
Oct 30 22:02:15.947804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:15.947213    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x" event={"ID":"7f637f6e-b239-4271-9822-fbdea237ccf5","Type":"ContainerStarted","Data":"c5a5eab0d8376b90d6b8bfaa8fa3b87a6b5d2135fb8a9e20aa444df2230b465e"}
Oct 30 22:02:17.010083 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.010041    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x" event={"ID":"7f637f6e-b239-4271-9822-fbdea237ccf5","Type":"ContainerStarted","Data":"8c327f5688b33e793423a45d2fb68f37f1cc4cc6fa8ab1e0da9d291ee2dddca7"}
Oct 30 22:02:17.184805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.182803    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x" podStartSLOduration=3.182758668 podStartE2EDuration="3.182758668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:17.13567317 +0000 UTC m=+1146.119580058" watchObservedRunningTime="2026-10-30 22:02:17.182758668 +0000 UTC m=+1146.166665544"
Oct 30 22:02:17.282855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.279270    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"]
Oct 30 22:02:17.294792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.294055    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.334690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.316103    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"]
Oct 30 22:02:17.387819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.362188    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.387819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.362238    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.387819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.362291    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftbp\" (UniqueName: \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.480804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.477095    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.480804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.477162    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.571861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.499682    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftbp\" (UniqueName: \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.604100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.508167    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.604519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.560904    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.900860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.784369    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.900860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.801979    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.900860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.787771    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftbp\" (UniqueName: \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\") pod \"maas-api-revoke-sub-a7fc7cab-fp788\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.900860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.852025    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.900860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.879958    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.918805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.905175    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:17.918805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.911631    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:17.981806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.939943    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:17.981806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.940046    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:17.983222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.983185    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:17.983435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:17.983399    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.112805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.109337    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:02:18.118392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.111600    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:02:18.146950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.146907    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.363544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.363502    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.377063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.377028    5510 generic.go:356] "Generic (PLEG): container finished" podID="7f637f6e-b239-4271-9822-fbdea237ccf5" containerID="8c327f5688b33e793423a45d2fb68f37f1cc4cc6fa8ab1e0da9d291ee2dddca7" exitCode=0
Oct 30 22:02:18.377491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.377462    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x" event={"ID":"7f637f6e-b239-4271-9822-fbdea237ccf5","Type":"ContainerDied","Data":"8c327f5688b33e793423a45d2fb68f37f1cc4cc6fa8ab1e0da9d291ee2dddca7"}
Oct 30 22:02:18.438356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.438251    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.452928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.452876    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.528924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.528884    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:18.529216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.529182    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.536956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.536864    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.564305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.564261    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:02:18.731809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.730132    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.731809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.730581    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:02:18.850820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.838252    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.850820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.838637    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:02:18.880816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.853414    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.880816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.878204    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:18.880816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.880536    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:18.897820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.881439    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:02:18.981805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:18.949542    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:19.126806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.126598    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:02:19.127279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.127107    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:02:19.273821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.269015    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:19.382059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.381890    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Oct 30 22:02:19.455232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.455007    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:19.464345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.462183    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:02:19.464345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.462246    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:02:19.464345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.462286    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:02:19.464345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.462308    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"]
Oct 30 22:02:19.476872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.476813    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:19.481138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.478445    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:02:19.488310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.488289    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:02:19.488481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.488469    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:02:19.489642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.489617    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:02:19.491591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.491576    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:02:19.530909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.525953    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788" event={"ID":"a314676d-b945-4a06-9b97-6a8907b0db2e","Type":"ContainerStarted","Data":"b91307818dc76dbf2d803638a13d91ae39f8c722bab0f86333e5b7bca26f73d0"}
Oct 30 22:02:19.577379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.576506    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:02:19.579239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.578246    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:02:19.579239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.579181    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:02:19.728149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.728119    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:19.788062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.788035    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\") pod \"7f637f6e-b239-4271-9822-fbdea237ccf5\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") "
Oct 30 22:02:19.788204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.788140    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\") pod \"7f637f6e-b239-4271-9822-fbdea237ccf5\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") "
Oct 30 22:02:19.788204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.788196    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\" (UniqueName: \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\") pod \"7f637f6e-b239-4271-9822-fbdea237ccf5\" (UID: \"7f637f6e-b239-4271-9822-fbdea237ccf5\") "
Oct 30 22:02:19.788445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.788408    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca" pod "7f637f6e-b239-4271-9822-fbdea237ccf5" (UID: "7f637f6e-b239-4271-9822-fbdea237ccf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:19.788815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.788757    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:19.790132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.790110    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca" pod "7f637f6e-b239-4271-9822-fbdea237ccf5" (UID: "7f637f6e-b239-4271-9822-fbdea237ccf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:19.790237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.790219    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc" pod "7f637f6e-b239-4271-9822-fbdea237ccf5" (UID: "7f637f6e-b239-4271-9822-fbdea237ccf5"). InnerVolumeSpecName "kube-api-access-8bvwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:19.890006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.889970    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f637f6e-b239-4271-9822-fbdea237ccf5-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:19.890006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:19.890004    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvwc\" (UniqueName: \"kubernetes.io/projected/7f637f6e-b239-4271-9822-fbdea237ccf5-kube-api-access-8bvwc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:20.545086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:20.545053    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"
Oct 30 22:02:20.545538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:20.545075    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x" event={"ID":"7f637f6e-b239-4271-9822-fbdea237ccf5","Type":"ContainerDied","Data":"c5a5eab0d8376b90d6b8bfaa8fa3b87a6b5d2135fb8a9e20aa444df2230b465e"}
Oct 30 22:02:20.545538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:20.545138    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5a5eab0d8376b90d6b8bfaa8fa3b87a6b5d2135fb8a9e20aa444df2230b465e"
Oct 30 22:02:20.547041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:20.547017    5510 generic.go:356] "Generic (PLEG): container finished" podID="a314676d-b945-4a06-9b97-6a8907b0db2e" containerID="264127c2700132114d63159236e863de6667a4fd046ae61ae58d383553c848b0" exitCode=0
Oct 30 22:02:20.547189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:20.547107    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788" event={"ID":"a314676d-b945-4a06-9b97-6a8907b0db2e","Type":"ContainerDied","Data":"264127c2700132114d63159236e863de6667a4fd046ae61ae58d383553c848b0"}
Oct 30 22:02:21.046385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.046353    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"]
Oct 30 22:02:21.050063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.050036    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f18ef5-qwj7v"]
Oct 30 22:02:21.126471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.126442    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2317202d-e739-4622-9aff-3b36b6f6d501" path="/var/lib/kubelet/pods/2317202d-e739-4622-9aff-3b36b6f6d501/volumes"
Oct 30 22:02:21.707069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.707043    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:21.729399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.729373    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\") pod \"a314676d-b945-4a06-9b97-6a8907b0db2e\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") "
Oct 30 22:02:21.729533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.729488    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\" (UniqueName: \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\") pod \"a314676d-b945-4a06-9b97-6a8907b0db2e\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") "
Oct 30 22:02:21.729533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.729520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\") pod \"a314676d-b945-4a06-9b97-6a8907b0db2e\" (UID: \"a314676d-b945-4a06-9b97-6a8907b0db2e\") "
Oct 30 22:02:21.729740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.729718    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca" pod "a314676d-b945-4a06-9b97-6a8907b0db2e" (UID: "a314676d-b945-4a06-9b97-6a8907b0db2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:21.730220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.730197    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:21.731599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.731574    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca" pod "a314676d-b945-4a06-9b97-6a8907b0db2e" (UID: "a314676d-b945-4a06-9b97-6a8907b0db2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:21.732229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.732201    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp" pod "a314676d-b945-4a06-9b97-6a8907b0db2e" (UID: "a314676d-b945-4a06-9b97-6a8907b0db2e"). InnerVolumeSpecName "kube-api-access-bftbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:21.831546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.831459    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a314676d-b945-4a06-9b97-6a8907b0db2e-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:21.831546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:21.831498    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftbp\" (UniqueName: \"kubernetes.io/projected/a314676d-b945-4a06-9b97-6a8907b0db2e-kube-api-access-bftbp\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:22.383653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:22.383550    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:22.574633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:22.574602    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"
Oct 30 22:02:22.574830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:22.574601    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788" event={"ID":"a314676d-b945-4a06-9b97-6a8907b0db2e","Type":"ContainerDied","Data":"b91307818dc76dbf2d803638a13d91ae39f8c722bab0f86333e5b7bca26f73d0"}
Oct 30 22:02:22.574830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:22.574742    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b91307818dc76dbf2d803638a13d91ae39f8c722bab0f86333e5b7bca26f73d0"
Oct 30 22:02:23.205000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:23.204965    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:23.363509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:23.363474    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:25.049036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.049003    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"]
Oct 30 22:02:25.095124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.095094    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f425ba-8tg67"]
Oct 30 22:02:25.128028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.127996    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be5295a-7bf4-41bc-8f4c-067c8c4f2d31" path="/var/lib/kubelet/pods/2be5295a-7bf4-41bc-8f4c-067c8c4f2d31/volumes"
Oct 30 22:02:25.664060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664016    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"]
Oct 30 22:02:25.664594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664579    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7f637f6e-b239-4271-9822-fbdea237ccf5" containerName="revoke-keys"
Oct 30 22:02:25.664594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664592    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f637f6e-b239-4271-9822-fbdea237ccf5" containerName="revoke-keys"
Oct 30 22:02:25.664736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664623    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a314676d-b945-4a06-9b97-6a8907b0db2e" containerName="revoke-keys"
Oct 30 22:02:25.664736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664628    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a314676d-b945-4a06-9b97-6a8907b0db2e" containerName="revoke-keys"
Oct 30 22:02:25.664736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664730    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f637f6e-b239-4271-9822-fbdea237ccf5" containerName="revoke-keys"
Oct 30 22:02:25.664947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.664742    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a314676d-b945-4a06-9b97-6a8907b0db2e" containerName="revoke-keys"
Oct 30 22:02:25.668769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.668749    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.721418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.721390    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"]
Oct 30 22:02:25.763050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.763022    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:25.763248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.763036    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:25.763248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.763079    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:02:25.763728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.763711    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:25.794339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.794310    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.794560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.794538    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xhc\" (UniqueName: \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.794617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.794591    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.896010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.895977    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xhc\" (UniqueName: \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.896479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.896448    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.896753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.896733    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.897891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.897868    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.901869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.900891    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.961050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.961019    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xhc\" (UniqueName: \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\") pod \"maas-api-revoke-sub-fc9b6907-5t4mp\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:25.982860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:25.982837    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:26.360043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:26.359892    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"]
Oct 30 22:02:26.363257 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:26.363228    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a439047_3880_4a10_ae86_71721fd634ee.slice/crio-2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be WatchSource:0}: Error finding container 2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be: Status 404 returned error can't find the container with id 2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be
Oct 30 22:02:26.607504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:26.607471    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp" event={"ID":"5a439047-3880-4a10-ae86-71721fd634ee","Type":"ContainerStarted","Data":"2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be"}
Oct 30 22:02:26.825022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:26.824989    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:27.616191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:27.616099    5510 generic.go:356] "Generic (PLEG): container finished" podID="5a439047-3880-4a10-ae86-71721fd634ee" containerID="88fbbb51e55bbad1bdb5b1122fcb11843634f66cdc3ddb2bd582d4b2cd324545" exitCode=0
Oct 30 22:02:27.616191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:27.616143    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp" event={"ID":"5a439047-3880-4a10-ae86-71721fd634ee","Type":"ContainerDied","Data":"88fbbb51e55bbad1bdb5b1122fcb11843634f66cdc3ddb2bd582d4b2cd324545"}
Oct 30 22:02:28.785034 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.785004    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:28.836728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.836699    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\") pod \"5a439047-3880-4a10-ae86-71721fd634ee\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") "
Oct 30 22:02:28.836915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.836800    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\" (UniqueName: \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\") pod \"5a439047-3880-4a10-ae86-71721fd634ee\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") "
Oct 30 22:02:28.836990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.836923    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\") pod \"5a439047-3880-4a10-ae86-71721fd634ee\" (UID: \"5a439047-3880-4a10-ae86-71721fd634ee\") "
Oct 30 22:02:28.837059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.837039    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca" pod "5a439047-3880-4a10-ae86-71721fd634ee" (UID: "5a439047-3880-4a10-ae86-71721fd634ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:28.837529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.837507    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.838940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.838909    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca" pod "5a439047-3880-4a10-ae86-71721fd634ee" (UID: "5a439047-3880-4a10-ae86-71721fd634ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:28.839388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.839011    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc" pod "5a439047-3880-4a10-ae86-71721fd634ee" (UID: "5a439047-3880-4a10-ae86-71721fd634ee"). InnerVolumeSpecName "kube-api-access-h6xhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:28.939676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.939592    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a439047-3880-4a10-ae86-71721fd634ee-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.939676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:28.939631    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xhc\" (UniqueName: \"kubernetes.io/projected/5a439047-3880-4a10-ae86-71721fd634ee-kube-api-access-h6xhc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:29.633477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:29.633441    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"
Oct 30 22:02:29.633477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:29.633455    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp" event={"ID":"5a439047-3880-4a10-ae86-71721fd634ee","Type":"ContainerDied","Data":"2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be"}
Oct 30 22:02:29.633687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:29.633511    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c942630eca8c46304061d36d1ebe6b113f1a7bf64cf9e093f76036c2c6e20be"
Oct 30 22:02:31.246940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:31.246901    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:32.045797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:32.045746    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"]
Oct 30 22:02:32.093648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:32.093612    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50380152-6qftg"]
Oct 30 22:02:33.125042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:33.125001    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03912648-c62f-48c3-a2a3-714393eb44e4" path="/var/lib/kubelet/pods/03912648-c62f-48c3-a2a3-714393eb44e4/volumes"
Oct 30 22:02:34.502528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.502085    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"]
Oct 30 22:02:34.505239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.505215    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5a439047-3880-4a10-ae86-71721fd634ee" containerName="revoke-keys"
Oct 30 22:02:34.505239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.505237    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a439047-3880-4a10-ae86-71721fd634ee" containerName="revoke-keys"
Oct 30 22:02:34.505412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.505332    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a439047-3880-4a10-ae86-71721fd634ee" containerName="revoke-keys"
Oct 30 22:02:34.509226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.509201    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.559607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.559579    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"]
Oct 30 22:02:34.601428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.601398    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:02:34.601428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.601398    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:34.601428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.601405    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:02:34.605827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.605808    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:02:34.616737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.616710    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.616939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.616919    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.617088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.617070    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjnr\" (UniqueName: \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.718643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.718609    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.718817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.718746    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjnr\" (UniqueName: \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.718886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.718853    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.719373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.719351    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.721117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.721096    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.772558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.772495    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjnr\" (UniqueName: \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\") pod \"maas-api-revoke-sub-ebb856e0-wg6fl\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:34.822584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:34.822560    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:35.203631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:35.203545    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"]
Oct 30 22:02:35.208368 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:35.208338    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219f9b2f_3ad8_4172_8bea_d032ce6680a9.slice/crio-8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f WatchSource:0}: Error finding container 8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f: Status 404 returned error can't find the container with id 8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f
Oct 30 22:02:35.685820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:35.685763    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl" event={"ID":"219f9b2f-3ad8-4172-8bea-d032ce6680a9","Type":"ContainerStarted","Data":"bb319eb8fcb4f1e662eded05c42c7b391d4f145c7d8faab671266d93932f57c0"}
Oct 30 22:02:35.697120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:35.697082    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl" event={"ID":"219f9b2f-3ad8-4172-8bea-d032ce6680a9","Type":"ContainerStarted","Data":"8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f"}
Oct 30 22:02:35.728876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:35.728764    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:35.813863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:35.813768    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl" podStartSLOduration=1.813748381 podStartE2EDuration="1.813748381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:35.810926397 +0000 UTC m=+1164.794833274" watchObservedRunningTime="2026-10-30 22:02:35.813748381 +0000 UTC m=+1164.797655260"
Oct 30 22:02:36.694926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:36.694891    5510 generic.go:356] "Generic (PLEG): container finished" podID="219f9b2f-3ad8-4172-8bea-d032ce6680a9" containerID="bb319eb8fcb4f1e662eded05c42c7b391d4f145c7d8faab671266d93932f57c0" exitCode=0
Oct 30 22:02:36.695328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:36.694981    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl" event={"ID":"219f9b2f-3ad8-4172-8bea-d032ce6680a9","Type":"ContainerDied","Data":"bb319eb8fcb4f1e662eded05c42c7b391d4f145c7d8faab671266d93932f57c0"}
Oct 30 22:02:37.865046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.865019    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:37.916686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.916657    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:02:37.964040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.964010    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\" (UniqueName: \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\") pod \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") "
Oct 30 22:02:37.964192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.964094    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\") pod \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") "
Oct 30 22:02:37.964192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.964163    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\") pod \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\" (UID: \"219f9b2f-3ad8-4172-8bea-d032ce6680a9\") "
Oct 30 22:02:37.964505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.964478    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca" pod "219f9b2f-3ad8-4172-8bea-d032ce6680a9" (UID: "219f9b2f-3ad8-4172-8bea-d032ce6680a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:37.964928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.964906    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:37.966220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.966196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca" pod "219f9b2f-3ad8-4172-8bea-d032ce6680a9" (UID: "219f9b2f-3ad8-4172-8bea-d032ce6680a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:37.966306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:37.966200    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr" pod "219f9b2f-3ad8-4172-8bea-d032ce6680a9" (UID: "219f9b2f-3ad8-4172-8bea-d032ce6680a9"). InnerVolumeSpecName "kube-api-access-5kjnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:38.066309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.066273    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjnr\" (UniqueName: \"kubernetes.io/projected/219f9b2f-3ad8-4172-8bea-d032ce6680a9-kube-api-access-5kjnr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.066309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.066309    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f9b2f-3ad8-4172-8bea-d032ce6680a9-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.461401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.461367    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:02:38.462078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.462058    5510 cpu_manager.go:414] "containerMap: removing container" podUID="219f9b2f-3ad8-4172-8bea-d032ce6680a9" containerName="revoke-keys"
Oct 30 22:02:38.462131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.462082    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219f9b2f-3ad8-4172-8bea-d032ce6680a9" containerName="revoke-keys"
Oct 30 22:02:38.462251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.462241    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="219f9b2f-3ad8-4172-8bea-d032ce6680a9" containerName="revoke-keys"
Oct 30 22:02:38.466118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.466097    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571260    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571314    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571408    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8xn\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571454    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571504    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571665    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.571806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.571716    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.572166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.572143    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.572305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.572185    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673201    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673239    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673265    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673283    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673345    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8xn\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673374    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673406    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673563    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673852    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.673967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.673944    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.674041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.674023    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.674132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.674110    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.674292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.674271    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.675584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.675557    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.675800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.675767    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.694729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.694694    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:02:38.716426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.715573    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl" event={"ID":"219f9b2f-3ad8-4172-8bea-d032ce6680a9","Type":"ContainerDied","Data":"8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f"}
Oct 30 22:02:38.716426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.715636    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b858d7d72592372cb1537f7a97e0ec9b2c9d52503ae91d37b50630b3a63927f"
Oct 30 22:02:38.716426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.715705    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"
Oct 30 22:02:38.764896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.764691    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:38.782829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:38.780567    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8xn\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:39.080187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.080083    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:39.336340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.336254    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:39.555821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.544625    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:02:39.555821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.552997    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.587853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.587240    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720399    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720473    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720553    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720613    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720666    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720734    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zkq\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720773    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720859    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.720915    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.830075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.729577    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.839928    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840382    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840449    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840527    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840590    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840637    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840704    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zkq\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840744    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840831    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.840882    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.841288    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.841620    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.843308    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.843376    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.843546    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.845900 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:39.845254    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73350b1b_bc3e_436e_ba55_fcf13fc7dd21.slice/crio-5ca618d4de19a7fd39647209d2709d533526b9f028fb1334533335ffe5a892fe WatchSource:0}: Error finding container 5ca618d4de19a7fd39647209d2709d533526b9f028fb1334533335ffe5a892fe: Status 404 returned error can't find the container with id 5ca618d4de19a7fd39647209d2709d533526b9f028fb1334533335ffe5a892fe
Oct 30 22:02:39.870802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.849877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.870802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.850421    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.870802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.864991    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.934488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.937139    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.937222    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.937265    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zkq\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942609    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942675    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7q7x\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942695    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942717    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942746    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942803    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942820    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942848    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:39.950839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:39.942893    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.142732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.142263    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:02:40.150764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.150714    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.150957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.150935    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151128    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7q7x\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151169    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151433    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151603    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151816    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.151977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.151853    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.171803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.170817    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.171803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.171399    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.235951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.207948    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.261893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.248011    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.442344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.433468    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.442344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.433620    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.444447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.444419    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.481423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.454546    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:40.481423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.454857    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.481423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.456037    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:40.481423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.458070    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.481423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.467345    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7q7x\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.767700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.767666    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:40.931580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.931540    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:02:40.973171 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:40.973128    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28488c09_4bf6_41ed_adbe_d19cb5cca34e.slice/crio-68ceb007cdee21aaa7a136af6fe8324a3a3ec8fac73c7e7be1b045d1479454b5 WatchSource:0}: Error finding container 68ceb007cdee21aaa7a136af6fe8324a3a3ec8fac73c7e7be1b045d1479454b5: Status 404 returned error can't find the container with id 68ceb007cdee21aaa7a136af6fe8324a3a3ec8fac73c7e7be1b045d1479454b5
Oct 30 22:02:40.976102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.976069    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:40.976204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.976143    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:40.976204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:40.976181    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:41.068308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.068275    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" event={"ID":"28488c09-4bf6-41ed-adbe-d19cb5cca34e","Type":"ContainerStarted","Data":"68ceb007cdee21aaa7a136af6fe8324a3a3ec8fac73c7e7be1b045d1479454b5"}
Oct 30 22:02:41.070989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.070958    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" event={"ID":"73350b1b-bc3e-436e-ba55-fcf13fc7dd21","Type":"ContainerStarted","Data":"86d9ac00ef450c769d9cf605ae4c76a40d9c629b83caacaee569f730eb085ead"}
Oct 30 22:02:41.071306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.071270    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" event={"ID":"73350b1b-bc3e-436e-ba55-fcf13fc7dd21","Type":"ContainerStarted","Data":"5ca618d4de19a7fd39647209d2709d533526b9f028fb1334533335ffe5a892fe"}
Oct 30 22:02:41.083337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.083308    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:41.083641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.083621    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:41.089887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.089845    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:41.190646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.190605    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:02:41.195181 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:41.195148    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1f327b_3206_4358_ae30_e7bfef61053a.slice/crio-ce51a2f7eed68ab4402ba571afa1affa313dbc3d2aafa39c9351a08ac4ab26cf WatchSource:0}: Error finding container ce51a2f7eed68ab4402ba571afa1affa313dbc3d2aafa39c9351a08ac4ab26cf: Status 404 returned error can't find the container with id ce51a2f7eed68ab4402ba571afa1affa313dbc3d2aafa39c9351a08ac4ab26cf
Oct 30 22:02:41.200679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.200643    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:41.200819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.200738    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:41.200886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.200818    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:41.231730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:41.231677    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podStartSLOduration=3.231658069 podStartE2EDuration="3.231658069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.192383366 +0000 UTC m=+1170.176290242" watchObservedRunningTime="2026-10-30 22:02:41.231658069 +0000 UTC m=+1170.215564948"
Oct 30 22:02:42.089814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.089112    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.089814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.089172    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:42.140580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.140550    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" event={"ID":"8c1f327b-3206-4358-ae30-e7bfef61053a","Type":"ContainerStarted","Data":"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"}
Oct 30 22:02:42.140835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.140817    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" event={"ID":"8c1f327b-3206-4358-ae30-e7bfef61053a","Type":"ContainerStarted","Data":"ce51a2f7eed68ab4402ba571afa1affa313dbc3d2aafa39c9351a08ac4ab26cf"}
Oct 30 22:02:42.142510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.142485    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" event={"ID":"28488c09-4bf6-41ed-adbe-d19cb5cca34e","Type":"ContainerStarted","Data":"ee2b2039225677ab6b38574840773e9b24445a0a985a0aa283011e49bb9d020c"}
Oct 30 22:02:42.252704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.252645    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podStartSLOduration=3.252627258 podStartE2EDuration="3.252627258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:42.249839053 +0000 UTC m=+1171.233745932" watchObservedRunningTime="2026-10-30 22:02:42.252627258 +0000 UTC m=+1171.236534135"
Oct 30 22:02:42.386862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.386336    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podStartSLOduration=3.386312474 podStartE2EDuration="3.386312474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:42.369578976 +0000 UTC m=+1171.353485853" watchObservedRunningTime="2026-10-30 22:02:42.386312474 +0000 UTC m=+1171.370219352"
Oct 30 22:02:42.463806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.461471    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:42.463806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.461605    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.463806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.461649    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:42.770932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.770838    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:42.775232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.774879    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:42.775232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:42.774951    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:43.091863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.080408    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.091863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.080466    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:43.383860 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:02:43.381989    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:02:43.474803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.460215    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.474803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.460277    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:43.769806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.768158    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:43.769806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:43.768220    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:44.104116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.104025    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.139588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.139538    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:44.473861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.472202    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.473861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.472261    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:44.778861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.772895    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:44.778861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:44.772966    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:45.087860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.087124    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.087860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.087187    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:45.261807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.258023    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:45.531805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.531035    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.531805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.531094    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:45.769207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.769165    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:45.769477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:45.769447    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:46.088815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.087864    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.088815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.087925    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:46.519895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.483928    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.519895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.484001    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:46.775863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.775537    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.775863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:46.775629    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:47.085858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.085180    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.085858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.085245    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 22:02:47.473867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.459116    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.473867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.459193    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 22:02:47.768340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.768243    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.768626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:47.768590    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 22:02:48.417383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:48.417349    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:48.418552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:48.418526    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:48.419619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:48.419593    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:02:49.025715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.025676    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:49.027382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.027359    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:49.028213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.028195    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:02:49.078068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.078034    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:49.079334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.079310    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:49.080247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.080225    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:02:49.814862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:49.814828    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:53.562172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.562097    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:02:53.568279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.568258    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.661967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.661936    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.661983    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662145    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662212    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxc6\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662250    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662347    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662514    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662557    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.662604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.662592    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.669114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.669085    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:02:53.764307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764275    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764316    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764397    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764446    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxc6\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764469    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764513    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764628    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764666    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.764706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.764698    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.765035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.765009    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.765268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.765235    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.765393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.765315    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.765393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.765379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.765651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.765630    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.766827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.766802    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.767138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.767119    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.819302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.819237    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.819435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.819385    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxc6\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:53.884713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:53.884679    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:54.048336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.048285    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"]
Oct 30 22:02:54.052320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.052293    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40b463-rpcmp"]
Oct 30 22:02:54.224625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.224594    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:54.346212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.345950    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:02:54.367807 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:02:54.348139    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c8a305_c1a8_4a0f_8d65_b707833c9d3c.slice/crio-3909adbbf3d940bd7693680e0458ceae859d5e41f01fa3c6729388c913a0eff6 WatchSource:0}: Error finding container 3909adbbf3d940bd7693680e0458ceae859d5e41f01fa3c6729388c913a0eff6: Status 404 returned error can't find the container with id 3909adbbf3d940bd7693680e0458ceae859d5e41f01fa3c6729388c913a0eff6
Oct 30 22:02:54.367807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.364382    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:54.367807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.364468    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:54.367807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.364506    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:02:54.393373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:54.393344    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" event={"ID":"97c8a305-c1a8-4a0f-8d65-b707833c9d3c","Type":"ContainerStarted","Data":"3909adbbf3d940bd7693680e0458ceae859d5e41f01fa3c6729388c913a0eff6"}
Oct 30 22:02:55.057501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.057437    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"]
Oct 30 22:02:55.064011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.063963    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02a51a84-qd8d9"]
Oct 30 22:02:55.126112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.126081    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc" path="/var/lib/kubelet/pods/35b14ec5-74f6-4ae3-b4a5-3c45ffb835fc/volumes"
Oct 30 22:02:55.126599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.126584    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1054b5-ab67-47ff-99bb-84382e3f1492" path="/var/lib/kubelet/pods/3e1054b5-ab67-47ff-99bb-84382e3f1492/volumes"
Oct 30 22:02:55.402137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.402052    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" event={"ID":"97c8a305-c1a8-4a0f-8d65-b707833c9d3c","Type":"ContainerStarted","Data":"4c7d28fa4186979bcdf3247eeb56f5247087fd59eaca8aa1b60f0f1dd7a1824f"}
Oct 30 22:02:55.516190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.516138    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podStartSLOduration=2.516124525 podStartE2EDuration="2.516124525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:55.513180536 +0000 UTC m=+1184.497087413" watchObservedRunningTime="2026-10-30 22:02:55.516124525 +0000 UTC m=+1184.500031401"
Oct 30 22:02:55.885044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.885010    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:02:55.886328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.886303    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:55.886387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:55.886355    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:02:56.885412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:56.885375    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:56.885885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:56.885444    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:02:57.885317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:57.885281    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:57.885507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:57.885333    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:02:58.718169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:58.718135    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:58.889873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:58.889613    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:58.889873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:58.889711    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:02:59.885233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:59.885194    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:02:59.885410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:02:59.885258    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:03:00.886124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:00.886084    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:03:00.886644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:00.886150    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 22:03:01.889214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:01.889181    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:03:01.889610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:01.889565    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:03:01.890272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:01.890251    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:03:03.226591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:03.226553    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:06.048381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:06.048343    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"]
Oct 30 22:03:06.096487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:06.096446    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98287cab-nlg9m"]
Oct 30 22:03:07.125841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:07.125793    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439eff63-80e5-4590-b5b4-049ba69e2ce0" path="/var/lib/kubelet/pods/439eff63-80e5-4590-b5b4-049ba69e2ce0/volumes"
Oct 30 22:03:07.741281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:07.741251    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:08.344218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.344172    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"]
Oct 30 22:03:08.349324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.349298    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.398218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.398187    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:08.403638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.403603    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"]
Oct 30 22:03:08.442853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.442822    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:08.443084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.442874    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:08.444097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.444073    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:08.481773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.481745    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.481975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.481954    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.482042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.482014    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8l4\" (UniqueName: \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.584042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.584002    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.584269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.584080    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8l4\" (UniqueName: \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.584269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.584224    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.585110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.585079    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.594560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.594498    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.637682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.637646    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8l4\" (UniqueName: \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\") pod \"maas-api-revoke-sub-a9db5747-qfmg7\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:08.665201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:08.665174    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:09.045805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:09.045742    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"]
Oct 30 22:03:09.051339 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:09.051303    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf298f672_77a7_42ad_9531_5066a2229c6b.slice/crio-c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0 WatchSource:0}: Error finding container c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0: Status 404 returned error can't find the container with id c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0
Oct 30 22:03:09.533604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:09.533568    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7" event={"ID":"f298f672-77a7-42ad-9531-5066a2229c6b","Type":"ContainerStarted","Data":"a26f611bca362387f4ac2004c9d96c522a9264f006858407fdfbc3cf3fca6e17"}
Oct 30 22:03:09.533989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:09.533621    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7" event={"ID":"f298f672-77a7-42ad-9531-5066a2229c6b","Type":"ContainerStarted","Data":"c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0"}
Oct 30 22:03:09.643231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:09.643138    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7" podStartSLOduration=1.6431257860000001 podStartE2EDuration="1.643125786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:09.640429649 +0000 UTC m=+1198.624336526" watchObservedRunningTime="2026-10-30 22:03:09.643125786 +0000 UTC m=+1198.627032662"
Oct 30 22:03:10.567806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:10.565881    5510 generic.go:356] "Generic (PLEG): container finished" podID="f298f672-77a7-42ad-9531-5066a2229c6b" containerID="a26f611bca362387f4ac2004c9d96c522a9264f006858407fdfbc3cf3fca6e17" exitCode=0
Oct 30 22:03:10.567806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:10.566095    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7" event={"ID":"f298f672-77a7-42ad-9531-5066a2229c6b","Type":"ContainerDied","Data":"a26f611bca362387f4ac2004c9d96c522a9264f006858407fdfbc3cf3fca6e17"}
Oct 30 22:03:11.388804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:11.386200    5510 scope.go:122] "RemoveContainer" containerID="7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3"
Oct 30 22:03:11.554048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:11.551052    5510 scope.go:122] "RemoveContainer" containerID="3bf6b28f20aed8a2f580f9b719c9090859ee891b64277d9ceb533788dda5a800"
Oct 30 22:03:11.574342 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:03:11.574310    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3\": container with ID starting with 7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3 not found: ID does not exist" containerID="7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3"
Oct 30 22:03:11.574864 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:03:11.574835    5510 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3\": container with ID starting with 7375692a3d37d6aeb88ff3eb52eb4daa091995bac06f3afad4872a89a46a55c3 not found: ID does not exist"
Oct 30 22:03:11.668391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:11.668324    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:03:11.705829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:11.699892    5510 scope.go:122] "RemoveContainer" containerID="68ba9afefb42cab8f11adf4bbd99355e22d3d9a8e827ab0df16aabe5d21cc7f0"
Oct 30 22:03:12.307526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.307500    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:12.312515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.312445    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:12.410816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.395693    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\" (UniqueName: \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\") pod \"f298f672-77a7-42ad-9531-5066a2229c6b\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") "
Oct 30 22:03:12.410816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.395797    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\") pod \"f298f672-77a7-42ad-9531-5066a2229c6b\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") "
Oct 30 22:03:12.410816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.396131    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\") pod \"f298f672-77a7-42ad-9531-5066a2229c6b\" (UID: \"f298f672-77a7-42ad-9531-5066a2229c6b\") "
Oct 30 22:03:12.411556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.411507    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca" pod "f298f672-77a7-42ad-9531-5066a2229c6b" (UID: "f298f672-77a7-42ad-9531-5066a2229c6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:12.428537 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.428264    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca" pod "f298f672-77a7-42ad-9531-5066a2229c6b" (UID: "f298f672-77a7-42ad-9531-5066a2229c6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:12.479930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.448704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4" pod "f298f672-77a7-42ad-9531-5066a2229c6b" (UID: "f298f672-77a7-42ad-9531-5066a2229c6b"). InnerVolumeSpecName "kube-api-access-sb8l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:12.502103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.500117    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:12.502103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.500149    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8l4\" (UniqueName: \"kubernetes.io/projected/f298f672-77a7-42ad-9531-5066a2229c6b-kube-api-access-sb8l4\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:12.502103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.500163    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f298f672-77a7-42ad-9531-5066a2229c6b-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:12.642132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.642050    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"
Oct 30 22:03:12.642132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.642076    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7" event={"ID":"f298f672-77a7-42ad-9531-5066a2229c6b","Type":"ContainerDied","Data":"c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0"}
Oct 30 22:03:12.642132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:12.642131    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c461724e1366eae79c679481b72880feadd1781db1ba4478c5fe7f7940acecc0"
Oct 30 22:03:13.054836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:13.054799    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"]
Oct 30 22:03:13.059837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:13.059766    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753513c8-6vc4j"]
Oct 30 22:03:13.126130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:13.126099    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27893681-7aff-4f5c-9fec-f062866698be" path="/var/lib/kubelet/pods/27893681-7aff-4f5c-9fec-f062866698be/volumes"
Oct 30 22:03:13.281278 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:03:13.281240    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:03:14.006127 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.006093    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"]
Oct 30 22:03:14.006704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.006691    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f298f672-77a7-42ad-9531-5066a2229c6b" containerName="revoke-keys"
Oct 30 22:03:14.006757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.006704    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f298f672-77a7-42ad-9531-5066a2229c6b" containerName="revoke-keys"
Oct 30 22:03:14.006849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.006839    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f298f672-77a7-42ad-9531-5066a2229c6b" containerName="revoke-keys"
Oct 30 22:03:14.012356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.011606    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.064900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.064868    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"]
Oct 30 22:03:14.105903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.105868    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:14.106072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.105972    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:14.106513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.106488    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:14.106615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.106512    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:14.121266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.121235    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.121365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.121292    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.121448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.121429    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8wt\" (UniqueName: \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.223493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.223463    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.223672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.223613    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.223747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.223689    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8wt\" (UniqueName: \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.224275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.224254    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.225897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.225871    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.277285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.277228    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8wt\" (UniqueName: \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\") pod \"maas-api-revoke-sub-c848b8c3-5zcrl\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.329720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.329692    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:14.707978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:14.707951    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"]
Oct 30 22:03:14.713773 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:14.713747    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7564101_1af7_44e7_80ab_acfe60e29f84.slice/crio-ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3 WatchSource:0}: Error finding container ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3: Status 404 returned error can't find the container with id ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3
Oct 30 22:03:15.666942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:15.666902    5510 generic.go:356] "Generic (PLEG): container finished" podID="d7564101-1af7-44e7-80ab-acfe60e29f84" containerID="ccd44990e52eedb1909cb52d44b5ef49e5c25186b0c934ffe05d1082409a3f74" exitCode=0
Oct 30 22:03:15.667316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:15.666989    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl" event={"ID":"d7564101-1af7-44e7-80ab-acfe60e29f84","Type":"ContainerDied","Data":"ccd44990e52eedb1909cb52d44b5ef49e5c25186b0c934ffe05d1082409a3f74"}
Oct 30 22:03:15.667316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:15.667048    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl" event={"ID":"d7564101-1af7-44e7-80ab-acfe60e29f84","Type":"ContainerStarted","Data":"ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3"}
Oct 30 22:03:16.218178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.218143    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"]
Oct 30 22:03:16.222614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.222593    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.275060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.275032    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"]
Oct 30 22:03:16.353390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.353355    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.353556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.353413    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.353598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.353566    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqzz\" (UniqueName: \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.454647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.454611    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.454845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.454741    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.454924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.454909    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqzz\" (UniqueName: \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.455463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.455437    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.456980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.456956    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.513272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.513207    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqzz\" (UniqueName: \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\") pod \"maas-api-revoke-sub-2a7968a8-bm9t4\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.536923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.536898    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:16.737826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.737760    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:16.926075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.926052    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:16.976226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:16.976195    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"]
Oct 30 22:03:16.981447 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:16.981414    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode705defa_1b51_4419_87d7_f954605fba90.slice/crio-31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3 WatchSource:0}: Error finding container 31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3: Status 404 returned error can't find the container with id 31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3
Oct 30 22:03:17.064729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.064700    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\") pod \"d7564101-1af7-44e7-80ab-acfe60e29f84\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") "
Oct 30 22:03:17.064870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.064745    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\" (UniqueName: \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\") pod \"d7564101-1af7-44e7-80ab-acfe60e29f84\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") "
Oct 30 22:03:17.064870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.064858    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\") pod \"d7564101-1af7-44e7-80ab-acfe60e29f84\" (UID: \"d7564101-1af7-44e7-80ab-acfe60e29f84\") "
Oct 30 22:03:17.065106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.065078    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca" pod "d7564101-1af7-44e7-80ab-acfe60e29f84" (UID: "d7564101-1af7-44e7-80ab-acfe60e29f84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:17.065488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.065472    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.066725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.066699    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca" pod "d7564101-1af7-44e7-80ab-acfe60e29f84" (UID: "d7564101-1af7-44e7-80ab-acfe60e29f84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:17.066865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.066733    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt" pod "d7564101-1af7-44e7-80ab-acfe60e29f84" (UID: "d7564101-1af7-44e7-80ab-acfe60e29f84"). InnerVolumeSpecName "kube-api-access-jw8wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:17.166452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.166386    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7564101-1af7-44e7-80ab-acfe60e29f84-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.166452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.166413    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8wt\" (UniqueName: \"kubernetes.io/projected/d7564101-1af7-44e7-80ab-acfe60e29f84-kube-api-access-jw8wt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:17.684220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.684192    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"
Oct 30 22:03:17.684415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.684238    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl" event={"ID":"d7564101-1af7-44e7-80ab-acfe60e29f84","Type":"ContainerDied","Data":"ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3"}
Oct 30 22:03:17.684415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.684295    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2022177393e3a159303666c0bef1c6b88eecfed98fd79f122ffe0ed4da30e3"
Oct 30 22:03:17.685898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.685876    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4" event={"ID":"e705defa-1b51-4419-87d7-f954605fba90","Type":"ContainerStarted","Data":"fb4f06a8aac75b0cd04879ba017fce3f6b797cdfa00a5ce63b44571844617f2e"}
Oct 30 22:03:17.686030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.685912    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4" event={"ID":"e705defa-1b51-4419-87d7-f954605fba90","Type":"ContainerStarted","Data":"31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3"}
Oct 30 22:03:17.795248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:17.795156    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4" podStartSLOduration=1.795139034 podStartE2EDuration="1.795139034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:17.793843013 +0000 UTC m=+1206.777749889" watchObservedRunningTime="2026-10-30 22:03:17.795139034 +0000 UTC m=+1206.779045909"
Oct 30 22:03:18.696840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:18.696808    5510 generic.go:356] "Generic (PLEG): container finished" podID="e705defa-1b51-4419-87d7-f954605fba90" containerID="fb4f06a8aac75b0cd04879ba017fce3f6b797cdfa00a5ce63b44571844617f2e" exitCode=0
Oct 30 22:03:18.697003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:18.696861    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4" event={"ID":"e705defa-1b51-4419-87d7-f954605fba90","Type":"ContainerDied","Data":"fb4f06a8aac75b0cd04879ba017fce3f6b797cdfa00a5ce63b44571844617f2e"}
Oct 30 22:03:19.863387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:19.863359    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:20.006481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.006452    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\") pod \"e705defa-1b51-4419-87d7-f954605fba90\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") "
Oct 30 22:03:20.006675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.006552    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\") pod \"e705defa-1b51-4419-87d7-f954605fba90\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") "
Oct 30 22:03:20.006675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.006581    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\" (UniqueName: \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\") pod \"e705defa-1b51-4419-87d7-f954605fba90\" (UID: \"e705defa-1b51-4419-87d7-f954605fba90\") "
Oct 30 22:03:20.006904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.006871    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca" pod "e705defa-1b51-4419-87d7-f954605fba90" (UID: "e705defa-1b51-4419-87d7-f954605fba90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:20.007237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.007215    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e705defa-1b51-4419-87d7-f954605fba90-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:20.009048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.009022    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca" pod "e705defa-1b51-4419-87d7-f954605fba90" (UID: "e705defa-1b51-4419-87d7-f954605fba90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:20.009666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.009645    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz" pod "e705defa-1b51-4419-87d7-f954605fba90" (UID: "e705defa-1b51-4419-87d7-f954605fba90"). InnerVolumeSpecName "kube-api-access-ksqzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:20.044498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.044473    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"]
Oct 30 22:03:20.048172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.048150    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ec762a-xgtws"]
Oct 30 22:03:20.108697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.108663    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e705defa-1b51-4419-87d7-f954605fba90-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:20.108697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.108696    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksqzz\" (UniqueName: \"kubernetes.io/projected/e705defa-1b51-4419-87d7-f954605fba90-kube-api-access-ksqzz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:20.711852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.711813    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4" event={"ID":"e705defa-1b51-4419-87d7-f954605fba90","Type":"ContainerDied","Data":"31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3"}
Oct 30 22:03:20.712024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.711877    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31823565646ccf140861658499963f9e55b1df0db11c5aec6754cca17f65b3c3"
Oct 30 22:03:20.712024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:20.711829    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"
Oct 30 22:03:21.047220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.047059    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"]
Oct 30 22:03:21.050994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.050964    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad7745c-5dj24"]
Oct 30 22:03:21.131399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.131364    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7015b4c8-3749-45f7-bede-21dd394df0ea" path="/var/lib/kubelet/pods/7015b4c8-3749-45f7-bede-21dd394df0ea/volumes"
Oct 30 22:03:21.131965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.131947    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e995749d-8eca-422a-a8e6-7d94cd3905e2" path="/var/lib/kubelet/pods/e995749d-8eca-422a-a8e6-7d94cd3905e2/volumes"
Oct 30 22:03:21.320035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.319919    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:21.579551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.579474    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"]
Oct 30 22:03:21.580171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580149    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d7564101-1af7-44e7-80ab-acfe60e29f84" containerName="revoke-keys"
Oct 30 22:03:21.580171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580167    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7564101-1af7-44e7-80ab-acfe60e29f84" containerName="revoke-keys"
Oct 30 22:03:21.580358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580188    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e705defa-1b51-4419-87d7-f954605fba90" containerName="revoke-keys"
Oct 30 22:03:21.580358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580193    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e705defa-1b51-4419-87d7-f954605fba90" containerName="revoke-keys"
Oct 30 22:03:21.580358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580323    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7564101-1af7-44e7-80ab-acfe60e29f84" containerName="revoke-keys"
Oct 30 22:03:21.580358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.580337    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e705defa-1b51-4419-87d7-f954605fba90" containerName="revoke-keys"
Oct 30 22:03:21.585651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.585632    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.639417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.639386    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"]
Oct 30 22:03:21.679141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.679095    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:21.679393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.679108    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:21.679393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.679307    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:21.680086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.680062    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:21.732007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.731976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.732175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.732020    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw844\" (UniqueName: \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.732175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.732145    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.829358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.829323    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"]
Oct 30 22:03:21.834024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.833961    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:21.834153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.834090    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.834153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.834129    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw844\" (UniqueName: \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.834274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.834192    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.834812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.834773    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.836486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.836464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.890105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.888381    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw844\" (UniqueName: \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\") pod \"maas-api-revoke-sub-ea873ee5-rstwf\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.890105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.888708    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"]
Oct 30 22:03:21.903303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.901859    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:21.935383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.935346    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:21.935578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.935476    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxc89\" (UniqueName: \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:21.935578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:21.935565    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.037654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.037615    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.037990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.037965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.038400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.038072    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxc89\" (UniqueName: \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.038741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.038714    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.040380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.040354    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.091233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.091164    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxc89\" (UniqueName: \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\") pod \"maas-api-revoke-sub-b4c8448e-pj4xw\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.160559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.160531    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:22.289976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.289929    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"]
Oct 30 22:03:22.295609 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:22.294826    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68467546_4aa5_4445_82df_375d470f7d04.slice/crio-c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20 WatchSource:0}: Error finding container c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20: Status 404 returned error can't find the container with id c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20
Oct 30 22:03:22.539207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.539166    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"]
Oct 30 22:03:22.544923 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:22.544896    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7331d156_e0f1_4d91_9363_7d5e31f8629c.slice/crio-5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee WatchSource:0}: Error finding container 5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee: Status 404 returned error can't find the container with id 5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee
Oct 30 22:03:22.728681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.728587    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf" event={"ID":"68467546-4aa5-4445-82df-375d470f7d04","Type":"ContainerStarted","Data":"531153a5796c8728614388c8c6d04ef929ddd6684b0759914e17c3f62288017d"}
Oct 30 22:03:22.728681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.728644    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf" event={"ID":"68467546-4aa5-4445-82df-375d470f7d04","Type":"ContainerStarted","Data":"c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20"}
Oct 30 22:03:22.729964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.729936    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw" event={"ID":"7331d156-e0f1-4d91-9363-7d5e31f8629c","Type":"ContainerStarted","Data":"5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee"}
Oct 30 22:03:22.834880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:22.834824    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf" podStartSLOduration=1.834803247 podStartE2EDuration="1.834803247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:22.834140753 +0000 UTC m=+1211.818047641" watchObservedRunningTime="2026-10-30 22:03:22.834803247 +0000 UTC m=+1211.818710127"
Oct 30 22:03:23.567578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:23.567463    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:23.740355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:23.740320    5510 generic.go:356] "Generic (PLEG): container finished" podID="68467546-4aa5-4445-82df-375d470f7d04" containerID="531153a5796c8728614388c8c6d04ef929ddd6684b0759914e17c3f62288017d" exitCode=0
Oct 30 22:03:23.740574 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:23.740382    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf" event={"ID":"68467546-4aa5-4445-82df-375d470f7d04","Type":"ContainerDied","Data":"531153a5796c8728614388c8c6d04ef929ddd6684b0759914e17c3f62288017d"}
Oct 30 22:03:23.742293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:23.742271    5510 generic.go:356] "Generic (PLEG): container finished" podID="7331d156-e0f1-4d91-9363-7d5e31f8629c" containerID="433777bb164149ebe8c5ef47104fdfb7445845d858b1160e806e7c008bdac23b" exitCode=0
Oct 30 22:03:23.742431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:23.742349    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw" event={"ID":"7331d156-e0f1-4d91-9363-7d5e31f8629c","Type":"ContainerDied","Data":"433777bb164149ebe8c5ef47104fdfb7445845d858b1160e806e7c008bdac23b"}
Oct 30 22:03:24.047727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.047696    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"]
Oct 30 22:03:24.051900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.051873    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35b9eb3b-nkxrb"]
Oct 30 22:03:24.929460 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.929425    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:24.973085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.973058    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:24.991286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.991259    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\") pod \"7331d156-e0f1-4d91-9363-7d5e31f8629c\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") "
Oct 30 22:03:24.991411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.991329    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\") pod \"7331d156-e0f1-4d91-9363-7d5e31f8629c\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") "
Oct 30 22:03:24.991582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.991563    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\" (UniqueName: \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\") pod \"7331d156-e0f1-4d91-9363-7d5e31f8629c\" (UID: \"7331d156-e0f1-4d91-9363-7d5e31f8629c\") "
Oct 30 22:03:24.991634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.991614    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca" pod "7331d156-e0f1-4d91-9363-7d5e31f8629c" (UID: "7331d156-e0f1-4d91-9363-7d5e31f8629c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:24.992330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.992301    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:24.993598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.993563    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca" pod "7331d156-e0f1-4d91-9363-7d5e31f8629c" (UID: "7331d156-e0f1-4d91-9363-7d5e31f8629c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:24.993692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:24.993615    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89" pod "7331d156-e0f1-4d91-9363-7d5e31f8629c" (UID: "7331d156-e0f1-4d91-9363-7d5e31f8629c"). InnerVolumeSpecName "kube-api-access-bxc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:25.093339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.093300    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\" (UniqueName: \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\") pod \"68467546-4aa5-4445-82df-375d470f7d04\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") "
Oct 30 22:03:25.093535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.093378    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\") pod \"68467546-4aa5-4445-82df-375d470f7d04\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") "
Oct 30 22:03:25.093625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.093603    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\") pod \"68467546-4aa5-4445-82df-375d470f7d04\" (UID: \"68467546-4aa5-4445-82df-375d470f7d04\") "
Oct 30 22:03:25.094194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.094154    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxc89\" (UniqueName: \"kubernetes.io/projected/7331d156-e0f1-4d91-9363-7d5e31f8629c-kube-api-access-bxc89\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:25.094194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.094184    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7331d156-e0f1-4d91-9363-7d5e31f8629c-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:25.094376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.094324    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca" pod "68467546-4aa5-4445-82df-375d470f7d04" (UID: "68467546-4aa5-4445-82df-375d470f7d04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:25.096155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.096122    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca" pod "68467546-4aa5-4445-82df-375d470f7d04" (UID: "68467546-4aa5-4445-82df-375d470f7d04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:25.096533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.096514    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844" pod "68467546-4aa5-4445-82df-375d470f7d04" (UID: "68467546-4aa5-4445-82df-375d470f7d04"). InnerVolumeSpecName "kube-api-access-xw844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:25.125364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.125338    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8432a3-a4c4-4257-b454-fee41b6a5cef" path="/var/lib/kubelet/pods/ec8432a3-a4c4-4257-b454-fee41b6a5cef/volumes"
Oct 30 22:03:25.196810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.196757    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68467546-4aa5-4445-82df-375d470f7d04-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:25.196810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.196805    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw844\" (UniqueName: \"kubernetes.io/projected/68467546-4aa5-4445-82df-375d470f7d04-kube-api-access-xw844\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:25.197107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.196823    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68467546-4aa5-4445-82df-375d470f7d04-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:25.628092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628060    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"]
Oct 30 22:03:25.628741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628717    5510 cpu_manager.go:414] "containerMap: removing container" podUID="68467546-4aa5-4445-82df-375d470f7d04" containerName="revoke-keys"
Oct 30 22:03:25.628741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628734    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68467546-4aa5-4445-82df-375d470f7d04" containerName="revoke-keys"
Oct 30 22:03:25.628923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628750    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7331d156-e0f1-4d91-9363-7d5e31f8629c" containerName="revoke-keys"
Oct 30 22:03:25.628923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628756    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7331d156-e0f1-4d91-9363-7d5e31f8629c" containerName="revoke-keys"
Oct 30 22:03:25.628923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628897    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="68467546-4aa5-4445-82df-375d470f7d04" containerName="revoke-keys"
Oct 30 22:03:25.628923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.628913    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7331d156-e0f1-4d91-9363-7d5e31f8629c" containerName="revoke-keys"
Oct 30 22:03:25.634158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.634140    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.687279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.687246    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"]
Oct 30 22:03:25.703797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.703751    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxsv\" (UniqueName: \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.703947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.703912    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.704135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.704114    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.759704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.759671    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"
Oct 30 22:03:25.759919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.759698    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf" event={"ID":"68467546-4aa5-4445-82df-375d470f7d04","Type":"ContainerDied","Data":"c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20"}
Oct 30 22:03:25.759919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.759753    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72cdea2c8acc37154326e75758001e4202c41c849698d140c0c64b5b3f5fc20"
Oct 30 22:03:25.761365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.761341    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw" event={"ID":"7331d156-e0f1-4d91-9363-7d5e31f8629c","Type":"ContainerDied","Data":"5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee"}
Oct 30 22:03:25.761472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.761375    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"
Oct 30 22:03:25.761472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.761414    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a1a02a0189999108bd8e58a060b1018dddb60cce25d3193121d69642ca10aee"
Oct 30 22:03:25.805241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.805214    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.805371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.805307    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxsv\" (UniqueName: \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.805371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.805353    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.805842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.805821    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.807613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.807596    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.838879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.838853    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:25.918805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.913836    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxsv\" (UniqueName: \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\") pod \"maas-api-revoke-sub-bc2e78cb-sjb8x\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:25.956813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:25.955078    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:26.399530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:26.399486    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"]
Oct 30 22:03:26.405333 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:26.405299    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f91442f_9ecf_4152_b46c_e90c2093a800.slice/crio-9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1 WatchSource:0}: Error finding container 9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1: Status 404 returned error can't find the container with id 9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1
Oct 30 22:03:26.773652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:26.773615    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x" event={"ID":"8f91442f-9ecf-4152-b46c-e90c2093a800","Type":"ContainerStarted","Data":"e2cfd80390b0558b979a8473a1123f96863bdd162d74edab42fcf130e9f10a15"}
Oct 30 22:03:26.773841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:26.773687    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x" event={"ID":"8f91442f-9ecf-4152-b46c-e90c2093a800","Type":"ContainerStarted","Data":"9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1"}
Oct 30 22:03:26.882204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:26.882145    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x" podStartSLOduration=1.8821314930000002 podStartE2EDuration="1.882131493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:26.879431025 +0000 UTC m=+1215.863337903" watchObservedRunningTime="2026-10-30 22:03:26.882131493 +0000 UTC m=+1215.866038401"
Oct 30 22:03:27.784118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:27.784079    5510 generic.go:356] "Generic (PLEG): container finished" podID="8f91442f-9ecf-4152-b46c-e90c2093a800" containerID="e2cfd80390b0558b979a8473a1123f96863bdd162d74edab42fcf130e9f10a15" exitCode=0
Oct 30 22:03:27.784625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:27.784118    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x" event={"ID":"8f91442f-9ecf-4152-b46c-e90c2093a800","Type":"ContainerDied","Data":"e2cfd80390b0558b979a8473a1123f96863bdd162d74edab42fcf130e9f10a15"}
Oct 30 22:03:28.947682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.947654    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:28.956018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.955237    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\" (UniqueName: \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\") pod \"8f91442f-9ecf-4152-b46c-e90c2093a800\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") "
Oct 30 22:03:28.956018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.955374    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\") pod \"8f91442f-9ecf-4152-b46c-e90c2093a800\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") "
Oct 30 22:03:28.956018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.955523    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\") pod \"8f91442f-9ecf-4152-b46c-e90c2093a800\" (UID: \"8f91442f-9ecf-4152-b46c-e90c2093a800\") "
Oct 30 22:03:28.956018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.955984    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca" pod "8f91442f-9ecf-4152-b46c-e90c2093a800" (UID: "8f91442f-9ecf-4152-b46c-e90c2093a800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:28.956315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.956191    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:28.958175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.958151    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv" pod "8f91442f-9ecf-4152-b46c-e90c2093a800" (UID: "8f91442f-9ecf-4152-b46c-e90c2093a800"). InnerVolumeSpecName "kube-api-access-9dxsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:28.959796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:28.959755    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca" pod "8f91442f-9ecf-4152-b46c-e90c2093a800" (UID: "8f91442f-9ecf-4152-b46c-e90c2093a800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:29.061039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:29.059672    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxsv\" (UniqueName: \"kubernetes.io/projected/8f91442f-9ecf-4152-b46c-e90c2093a800-kube-api-access-9dxsv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:29.061039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:29.059706    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f91442f-9ecf-4152-b46c-e90c2093a800-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:29.802895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:29.802863    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"
Oct 30 22:03:29.802895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:29.802884    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x" event={"ID":"8f91442f-9ecf-4152-b46c-e90c2093a800","Type":"ContainerDied","Data":"9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1"}
Oct 30 22:03:29.803098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:29.802931    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f869ff0fe9db306022ba03f120471317626facd2e68e7d3e4121799e28b01b1"
Oct 30 22:03:30.049527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:30.049492    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"]
Oct 30 22:03:30.054119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:30.054054    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3716a776-2vd8c"]
Oct 30 22:03:30.209353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:30.209323    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:31.099581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:31.099547    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:31.127723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:31.127688    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bf61af-6adf-4c07-b2b9-b22c1bd947e1" path="/var/lib/kubelet/pods/e8bf61af-6adf-4c07-b2b9-b22c1bd947e1/volumes"
Oct 30 22:03:31.249665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:31.249612    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:31.699351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:31.699316    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:03:34.068191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.068155    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"]
Oct 30 22:03:34.123383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.123349    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"]
Oct 30 22:03:34.135832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.127938    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.145889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.142248    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7632c2d8-x42n7"]
Oct 30 22:03:34.167804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.166756    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1dd44a5-4hxkw"]
Oct 30 22:03:34.466972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.466915    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.489964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.489917    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.490597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.490560    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.491102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.491077    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.500194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.499858    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.500194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.500145    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.583806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.580909    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:03:34.583806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.581324    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.583806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.582755    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:34.855039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:34.854933    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.136414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.136302    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.172311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.172273    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31df64b2-218c-4d93-9ccd-17c2ca0ebb94" path="/var/lib/kubelet/pods/31df64b2-218c-4d93-9ccd-17c2ca0ebb94/volumes"
Oct 30 22:03:35.174310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.174287    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22ad7ae-4a49-429c-9063-a2256cac5d45" path="/var/lib/kubelet/pods/c22ad7ae-4a49-429c-9063-a2256cac5d45/volumes"
Oct 30 22:03:35.184710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.184688    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:03:35.503651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.499941    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.503651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.500450    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.521865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.518159    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:03:35.521865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.518231    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:03:35.551853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.531920    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:03:35.551853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.532028    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.551853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.532445    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:03:35.551853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.532841    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.551853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.533738    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:03:35.643836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.596446    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.660804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.596509    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.693816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.597666    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.597695    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.597714    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.646369    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.808871    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.808914    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.808950    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.808984    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:03:35.827885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:35.653738    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:36.169861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.132678    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 22:03:36.441925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.324373    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.522940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.516870    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.522940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.517176    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.535243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.535197    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.536447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.535444    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.660609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.660566    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.736807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.727684    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.821205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.819972    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:36.821205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.821141    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:03:36.855532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:36.854914    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.424573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.424365    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:03:37.436212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.436171    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.529811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.524237    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.530347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.525848    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.577991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.577942    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.749797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.725529    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.754796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.750090    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.892800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.889280    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:37.961810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.950762    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:03:37.973100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:37.973049    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:03:38.177649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.177564    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:03:38.186064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.184891    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:03:38.219332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.219299    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:03:38.232983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.232951    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:03:38.253799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.253755    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 22:03:38.254432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.254403    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" containerID="cri-o://b70faa79210f5ca5feee10b8e2a7048ee68f0ea048b512c24c30bc71ed203f94" gracePeriod=30
Oct 30 22:03:38.264862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.264366    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:03:38.268840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.267278    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:03:38.269516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.269240    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:03:38.276218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.271542    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:03:38.277126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.277096    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:03:38.293868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.293830    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 22:03:38.294110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.294066    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy" containerID="cri-o://225bcf12356246798a18d8235c593557fb987c81fc3b0e567a850097501d4c1f" gracePeriod=30
Oct 30 22:03:38.313802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.297894    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 22:03:38.313802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.298117    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" containerID="cri-o://860aedf10bead954eaa331f67d4ce6144fd8d00e233b41084e9b02920acebaa5" gracePeriod=30
Oct 30 22:03:38.313802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.299754    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 22:03:38.313802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:38.299904    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" containerID="cri-o://a471c7d92fda8fe394237828fa2afe9f1bc874dbc65e360a2bf8376dce9ad636" gracePeriod=30
Oct 30 22:03:39.275660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.275622    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:03:39.276642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.276621    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8f91442f-9ecf-4152-b46c-e90c2093a800" containerName="revoke-keys"
Oct 30 22:03:39.276704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.276646    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f91442f-9ecf-4152-b46c-e90c2093a800" containerName="revoke-keys"
Oct 30 22:03:39.276829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.276817    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f91442f-9ecf-4152-b46c-e90c2093a800" containerName="revoke-keys"
Oct 30 22:03:39.282836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.282811    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.337327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.337291    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:03:39.354639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.354606    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x978\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.354845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.354663    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.354957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.354916    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355044    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355102    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355172    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355233    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355269    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.355364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.355329    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457230    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457363    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457423    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457474    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457545    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457605    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457741    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x978\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457805    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.457910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.457842    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.458192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.458164    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.458338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.458307    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.458545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.458526    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.458625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.458602    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.460351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.460328    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.460678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.460657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.510921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.510890    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.511626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.511606    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x978\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\") pod \"e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") " pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:39.600761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:39.600660    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:40.064743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.064707    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:03:40.069837 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:40.069751    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93f619e_ee34_4897_a528_9a6971e23e24.slice/crio-a6fc24813fb3f92388b9579baf4f13c7a8251380afd6a1be5a359878fb627aeb WatchSource:0}: Error finding container a6fc24813fb3f92388b9579baf4f13c7a8251380afd6a1be5a359878fb627aeb: Status 404 returned error can't find the container with id a6fc24813fb3f92388b9579baf4f13c7a8251380afd6a1be5a359878fb627aeb
Oct 30 22:03:40.073183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.073129    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:40.073466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.073443    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:40.073643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.073624    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:40.126873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.126849    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:40.299365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.299326    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:03:40.305614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.305590    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.369774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.369741    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:03:40.381825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.381796    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.382106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.382084    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.382234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.382220    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.382508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.382477    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbgr\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.382724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.382619    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.383097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.383074    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.383174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.383127    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.383174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.383166    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.383273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.383199    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.439271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.439239    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" event={"ID":"d93f619e-ee34-4897-a528-9a6971e23e24","Type":"ContainerStarted","Data":"5dbedc6caec1c18ba3a045f41618ab25d39c16b00e36057f393e916205662c96"}
Oct 30 22:03:40.439427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.439298    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" event={"ID":"d93f619e-ee34-4897-a528-9a6971e23e24","Type":"ContainerStarted","Data":"a6fc24813fb3f92388b9579baf4f13c7a8251380afd6a1be5a359878fb627aeb"}
Oct 30 22:03:40.484901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.484861    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.484920    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.484955    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.484989    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485081    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485162    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485193    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485302    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbgr\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485312    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485442    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.485874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.485846    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.486124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.486098    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.486311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.486209    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.486946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.486921    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.488450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.488426    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.488624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.488608    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.601671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.601585    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:40.659165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.659103    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podStartSLOduration=1.659082631 podStartE2EDuration="1.659082631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:40.658021873 +0000 UTC m=+1229.641928748" watchObservedRunningTime="2026-10-30 22:03:40.659082631 +0000 UTC m=+1229.642989508"
Oct 30 22:03:40.660814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.660772    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.664270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.664240    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbgr\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:40.878323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.878238    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wz9xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:03:40.878323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.878307    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:40.929699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:40.929669    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:41.381795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.381736    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:03:41.393931 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:41.393901    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311e7f7f_676b_455b_bbc4_cf3b4dd972e8.slice/crio-0bcb0720992c00db73861f607e697e3052f94bb50dac9b3f2c344d4075189396 WatchSource:0}: Error finding container 0bcb0720992c00db73861f607e697e3052f94bb50dac9b3f2c344d4075189396: Status 404 returned error can't find the container with id 0bcb0720992c00db73861f607e697e3052f94bb50dac9b3f2c344d4075189396
Oct 30 22:03:41.397006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.396973    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:41.397086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.397040    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:41.397086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.397071    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:41.450689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.450662    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" event={"ID":"311e7f7f-676b-455b-bbc4-cf3b4dd972e8","Type":"ContainerStarted","Data":"0bcb0720992c00db73861f607e697e3052f94bb50dac9b3f2c344d4075189396"}
Oct 30 22:03:41.603210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.603167    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:41.603385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:41.603232    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:42.520803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.520135    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" event={"ID":"311e7f7f-676b-455b-bbc4-cf3b4dd972e8","Type":"ContainerStarted","Data":"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"}
Oct 30 22:03:42.601803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.601290    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:42.601803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.601385    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:42.719837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.718749    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podStartSLOduration=2.71871294 podStartE2EDuration="2.71871294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:42.670261047 +0000 UTC m=+1231.654167923" watchObservedRunningTime="2026-10-30 22:03:42.71871294 +0000 UTC m=+1231.702619817"
Oct 30 22:03:42.929941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.929851    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:42.931200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.931175    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:42.931327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:42.931226    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:43.089803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.078856    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mbkwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:03:43.089803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.078924    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:43.413178 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:03:43.413107    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:03:43.618861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.613200    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:43.618861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.613264    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:43.624801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.621857    5510 generic.go:356] "Generic (PLEG): container finished" podID="27964505-3e9a-4182-b996-db510769ff08" containerID="860aedf10bead954eaa331f67d4ce6144fd8d00e233b41084e9b02920acebaa5" exitCode=0
Oct 30 22:03:43.624801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.621965    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" event={"ID":"27964505-3e9a-4182-b996-db510769ff08","Type":"ContainerDied","Data":"860aedf10bead954eaa331f67d4ce6144fd8d00e233b41084e9b02920acebaa5"}
Oct 30 22:03:43.624801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.624139    5510 generic.go:356] "Generic (PLEG): container finished" podID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerID="b70faa79210f5ca5feee10b8e2a7048ee68f0ea048b512c24c30bc71ed203f94" exitCode=0
Oct 30 22:03:43.624801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.624248    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" event={"ID":"c0d6928a-1ea2-4c08-9760-627e4124e280","Type":"ContainerDied","Data":"b70faa79210f5ca5feee10b8e2a7048ee68f0ea048b512c24c30bc71ed203f94"}
Oct 30 22:03:43.627315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.626382    5510 generic.go:356] "Generic (PLEG): container finished" podID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerID="a471c7d92fda8fe394237828fa2afe9f1bc874dbc65e360a2bf8376dce9ad636" exitCode=0
Oct 30 22:03:43.627315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.626536    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" event={"ID":"3d5bd04a-318a-4316-9f9f-425fd4aaca05","Type":"ContainerDied","Data":"a471c7d92fda8fe394237828fa2afe9f1bc874dbc65e360a2bf8376dce9ad636"}
Oct 30 22:03:43.628619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.628587    5510 generic.go:356] "Generic (PLEG): container finished" podID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerID="225bcf12356246798a18d8235c593557fb987c81fc3b0e567a850097501d4c1f" exitCode=0
Oct 30 22:03:43.628744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.628653    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" event={"ID":"e9c22ae2-d03d-4c08-a65b-8f866b30869b","Type":"ContainerDied","Data":"225bcf12356246798a18d8235c593557fb987c81fc3b0e567a850097501d4c1f"}
Oct 30 22:03:43.716807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.716425    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-v4jj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 22:03:43.716807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.716495    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 22:03:43.932720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.932569    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:43.932720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.932653    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:43.949351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.947746    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 22:03:43.950050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:43.949963    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 22:03:44.046364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.046331    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.046642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.046617    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.046774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.046759    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.047106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.047253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047237    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.047374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047359    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.047493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.047617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047600    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.047835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047814    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.048022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.047999    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.048144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048129    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.048293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048278    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.048414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.048535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.048696 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048675    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.048857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.048833    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.049077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.049061    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\") pod \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\" (UID: \"3d5bd04a-318a-4316-9f9f-425fd4aaca05\") "
Oct 30 22:03:44.049230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.049213    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\") pod \"c0d6928a-1ea2-4c08-9760-627e4124e280\" (UID: \"c0d6928a-1ea2-4c08-9760-627e4124e280\") "
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.046759    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.050597    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.050913    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.051148    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.051319    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.051336    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.073049    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.073286    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.073448    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.073817    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.073926    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "kube-api-access-klsdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.080076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.080017    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.103050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.101953    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:44.103050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.102879    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103872    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103913    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103930    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103944    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103958    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103974    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.103988    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104006    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104020    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6928a-1ea2-4c08-9760-627e4124e280-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104033    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104045    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104060    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104074    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.104121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.104090    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsdk\" (UniqueName: \"kubernetes.io/projected/c0d6928a-1ea2-4c08-9760-627e4124e280-kube-api-access-klsdk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.134481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.134399    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy" pod "c0d6928a-1ea2-4c08-9760-627e4124e280" (UID: "c0d6928a-1ea2-4c08-9760-627e4124e280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.134481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.134428    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.134705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.134541    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.137109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.137078    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5" pod "3d5bd04a-318a-4316-9f9f-425fd4aaca05" (UID: "3d5bd04a-318a-4316-9f9f-425fd4aaca05"). InnerVolumeSpecName "kube-api-access-w5xb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.173740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.173665    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 22:03:44.182966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.182939    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 22:03:44.225387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.225352    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5xb5\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-kube-api-access-w5xb5\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.225387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.225385    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.225655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.225398    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5bd04a-318a-4316-9f9f-425fd4aaca05-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.225655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.225412    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6928a-1ea2-4c08-9760-627e4124e280-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328224    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328352    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328390    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328442    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328496    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328537    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328569    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328607    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328634    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328644    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328662    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328673    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328698    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328794    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328862    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328929    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328940    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.328969    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.329054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329064    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329116    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\") pod \"27964505-3e9a-4182-b996-db510769ff08\" (UID: \"27964505-3e9a-4182-b996-db510769ff08\") "
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\") pod \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\" (UID: \"e9c22ae2-d03d-4c08-a65b-8f866b30869b\") "
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329374    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329567    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.329879    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330017    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330111    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330137    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330157    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330172    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330186    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330202    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27964505-3e9a-4182-b996-db510769ff08-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.330987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.330962    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.331179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.331158    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.335554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.335528    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.335653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.335589    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.336988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.336898    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:44.336988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.336918    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.337113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.337033    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.337113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.337088    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:44.337643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.337563    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:44.339128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.338877    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9" pod "e9c22ae2-d03d-4c08-a65b-8f866b30869b" (UID: "e9c22ae2-d03d-4c08-a65b-8f866b30869b"). InnerVolumeSpecName "kube-api-access-xxvm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.341828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.341796    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw" pod "27964505-3e9a-4182-b996-db510769ff08" (UID: "27964505-3e9a-4182-b996-db510769ff08"). InnerVolumeSpecName "kube-api-access-pqqjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451008    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451045    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451055    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451064    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqqjw\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-kube-api-access-pqqjw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451074    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27964505-3e9a-4182-b996-db510769ff08-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451082    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451090    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451100    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27964505-3e9a-4182-b996-db510769ff08-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451108    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451116    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvm9\" (UniqueName: \"kubernetes.io/projected/e9c22ae2-d03d-4c08-a65b-8f866b30869b-kube-api-access-xxvm9\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451126    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27964505-3e9a-4182-b996-db510769ff08-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.451675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.451136    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c22ae2-d03d-4c08-a65b-8f866b30869b-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.636708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.636633    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:44.637756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.637354    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:44.648411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.648378    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6" event={"ID":"27964505-3e9a-4182-b996-db510769ff08","Type":"ContainerDied","Data":"d33739c4ef73a3995999ce09159671c48927aabc74e2321980895fd24143e1f8"}
Oct 30 22:03:44.648513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.648452    5510 scope.go:122] "RemoveContainer" containerID="860aedf10bead954eaa331f67d4ce6144fd8d00e233b41084e9b02920acebaa5"
Oct 30 22:03:44.648629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.648614    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"
Oct 30 22:03:44.741931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.733393    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh" event={"ID":"c0d6928a-1ea2-4c08-9760-627e4124e280","Type":"ContainerDied","Data":"6ecac4b684a29ad8c9ded64092aad54452bf8bbdd65be258ec8b4f9ee3a25426"}
Oct 30 22:03:44.741931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.733523    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"
Oct 30 22:03:44.741931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.736490    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp" event={"ID":"3d5bd04a-318a-4316-9f9f-425fd4aaca05","Type":"ContainerDied","Data":"47be1c9a222e64d223bcd02d5695c53ddcfef95a070dcaa8923b5175de4b2ece"}
Oct 30 22:03:44.741931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.736583    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"
Oct 30 22:03:44.827889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.827865    5510 scope.go:122] "RemoveContainer" containerID="b70faa79210f5ca5feee10b8e2a7048ee68f0ea048b512c24c30bc71ed203f94"
Oct 30 22:03:44.829020 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.828996    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557" event={"ID":"e9c22ae2-d03d-4c08-a65b-8f866b30869b","Type":"ContainerDied","Data":"8b02dc2ecd7189d0860dedfb92880fa8785072c37a9479131bbf586d0e12bcc8"}
Oct 30 22:03:44.829512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.829496    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"
Oct 30 22:03:44.842418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.842389    5510 scope.go:122] "RemoveContainer" containerID="a471c7d92fda8fe394237828fa2afe9f1bc874dbc65e360a2bf8376dce9ad636"
Oct 30 22:03:44.851646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.851631    5510 scope.go:122] "RemoveContainer" containerID="225bcf12356246798a18d8235c593557fb987c81fc3b0e567a850097501d4c1f"
Oct 30 22:03:44.931031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.930997    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:44.931180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.931059    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:44.973968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:44.973857    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:45.035854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.034827    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 22:03:45.047376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.047325    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-v4jj6"]
Oct 30 22:03:45.200806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.198043    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27964505-3e9a-4182-b996-db510769ff08" path="/var/lib/kubelet/pods/27964505-3e9a-4182-b996-db510769ff08/volumes"
Oct 30 22:03:45.200806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.198691    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 22:03:45.200806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.198730    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-27557"]
Oct 30 22:03:45.268489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.268425    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 22:03:45.281992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.281957    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wz9xp"]
Oct 30 22:03:45.389924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.389834    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 22:03:45.402799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.394740    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mbkwh"]
Oct 30 22:03:45.606321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.606279    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:45.606646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.606599    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:45.930105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.930056    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:45.930611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:45.930116    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:46.601483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:46.601441    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 22:03:46.601662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:46.601499    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 22:03:46.930515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:46.930420    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:46.930515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:46.930479    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:47.050122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.050085    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"]
Oct 30 22:03:47.086569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.086527    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7175e3-tj6pm"]
Oct 30 22:03:47.132178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.132134    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263488a1-3a03-4a71-adf5-1817370769b3" path="/var/lib/kubelet/pods/263488a1-3a03-4a71-adf5-1817370769b3/volumes"
Oct 30 22:03:47.132965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.132942    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" path="/var/lib/kubelet/pods/3d5bd04a-318a-4316-9f9f-425fd4aaca05/volumes"
Oct 30 22:03:47.133544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.133525    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" path="/var/lib/kubelet/pods/c0d6928a-1ea2-4c08-9760-627e4124e280/volumes"
Oct 30 22:03:47.134632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.134610    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" path="/var/lib/kubelet/pods/e9c22ae2-d03d-4c08-a65b-8f866b30869b/volumes"
Oct 30 22:03:47.930883 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.930844    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:03:47.931372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:47.930910    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 22:03:48.054388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.054356    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:48.054850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.054820    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:48.055676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.055638    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:03:48.269008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.268974    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"]
Oct 30 22:03:48.270004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.269977    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270003    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270039    5510 cpu_manager.go:414] "containerMap: removing container" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270048    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270071    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270085    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270126    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy"
Oct 30 22:03:48.270156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270134    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy"
Oct 30 22:03:48.270480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270324    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5bd04a-318a-4316-9f9f-425fd4aaca05" containerName="istio-proxy"
Oct 30 22:03:48.270480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270343    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="27964505-3e9a-4182-b996-db510769ff08" containerName="istio-proxy"
Oct 30 22:03:48.270480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270355    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d6928a-1ea2-4c08-9760-627e4124e280" containerName="istio-proxy"
Oct 30 22:03:48.270480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.270370    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c22ae2-d03d-4c08-a65b-8f866b30869b" containerName="istio-proxy"
Oct 30 22:03:48.278051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.277840    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"]
Oct 30 22:03:48.278051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.277982    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.307986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.307957    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"]
Oct 30 22:03:48.312487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.312466    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.327958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.327930    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:48.331081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.331056    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:48.331909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.331766    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:48.344060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.332196    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:48.365896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.365836    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"]
Oct 30 22:03:48.414747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.414708    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.415121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.415097    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.415304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.415279    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.415473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.415378    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.415473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.415428    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv69q\" (UniqueName: \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.415642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.415627    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqvj\" (UniqueName: \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.517621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.517567    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqvj\" (UniqueName: \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.517910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.517889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.517999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.517965    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.518075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.518012    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.518075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.518063    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.518172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.518105    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv69q\" (UniqueName: \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.518935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.518908    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.519090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.518908    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.520682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.520652    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.520815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.520747    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.570873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.570840    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqvj\" (UniqueName: \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\") pod \"maas-api-revoke-sub-603a71df-kftlh\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.570984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.570966    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv69q\" (UniqueName: \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\") pod \"maas-api-revoke-sub-282dc4d9-x9b5l\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.610300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.610270    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:48.633588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.633543    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:48.992160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:48.992128    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"]
Oct 30 22:03:48.998612 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:48.998585    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beb9e4a_e922_4ae2_bccc_77039a16dd44.slice/crio-7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc WatchSource:0}: Error finding container 7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc: Status 404 returned error can't find the container with id 7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc
Oct 30 22:03:49.058202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.058161    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"]
Oct 30 22:03:49.066359 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:49.066330    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68c5da5_3b11_4a61_899f_c518d2a3820f.slice/crio-ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1 WatchSource:0}: Error finding container ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1: Status 404 returned error can't find the container with id ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1
Oct 30 22:03:49.217507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.216570    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:49.217507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.217504    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:49.218724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.218703    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:03:49.594626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.594596    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:49.879975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.879939    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l" event={"ID":"9beb9e4a-e922-4ae2-bccc-77039a16dd44","Type":"ContainerStarted","Data":"bae9d3f438874fc6a9adf66e3c7d6a203caac979be8b0b6e0b184fb164d1439a"}
Oct 30 22:03:49.880160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.880004    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l" event={"ID":"9beb9e4a-e922-4ae2-bccc-77039a16dd44","Type":"ContainerStarted","Data":"7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc"}
Oct 30 22:03:49.881654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.881628    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh" event={"ID":"b68c5da5-3b11-4a61-899f-c518d2a3820f","Type":"ContainerStarted","Data":"a6cab293f0011faec2fada4317d080d3f98b3bd73ce097780008961fb97fcc0a"}
Oct 30 22:03:49.881759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.881671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh" event={"ID":"b68c5da5-3b11-4a61-899f-c518d2a3820f","Type":"ContainerStarted","Data":"ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1"}
Oct 30 22:03:49.991950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:49.991900    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l" podStartSLOduration=1.991886315 podStartE2EDuration="1.991886315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:49.989624432 +0000 UTC m=+1238.973531303" watchObservedRunningTime="2026-10-30 22:03:49.991886315 +0000 UTC m=+1238.975793192"
Oct 30 22:03:50.098344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:50.098291    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh" podStartSLOduration=2.098274962 podStartE2EDuration="2.098274962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:50.096553829 +0000 UTC m=+1239.080460703" watchObservedRunningTime="2026-10-30 22:03:50.098274962 +0000 UTC m=+1239.082181839"
Oct 30 22:03:50.890344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:50.890309    5510 generic.go:356] "Generic (PLEG): container finished" podID="9beb9e4a-e922-4ae2-bccc-77039a16dd44" containerID="bae9d3f438874fc6a9adf66e3c7d6a203caac979be8b0b6e0b184fb164d1439a" exitCode=0
Oct 30 22:03:50.890620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:50.890352    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l" event={"ID":"9beb9e4a-e922-4ae2-bccc-77039a16dd44","Type":"ContainerDied","Data":"bae9d3f438874fc6a9adf66e3c7d6a203caac979be8b0b6e0b184fb164d1439a"}
Oct 30 22:03:50.892105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:50.892083    5510 generic.go:356] "Generic (PLEG): container finished" podID="b68c5da5-3b11-4a61-899f-c518d2a3820f" containerID="a6cab293f0011faec2fada4317d080d3f98b3bd73ce097780008961fb97fcc0a" exitCode=0
Oct 30 22:03:50.892225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:50.892166    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh" event={"ID":"b68c5da5-3b11-4a61-899f-c518d2a3820f","Type":"ContainerDied","Data":"a6cab293f0011faec2fada4317d080d3f98b3bd73ce097780008961fb97fcc0a"}
Oct 30 22:03:51.997288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:51.994911    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:52.111927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.111902    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:52.115608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.115584    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:52.198243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198164    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\") pod \"b68c5da5-3b11-4a61-899f-c518d2a3820f\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") "
Oct 30 22:03:52.198243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198241    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\" (UniqueName: \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\") pod \"b68c5da5-3b11-4a61-899f-c518d2a3820f\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") "
Oct 30 22:03:52.198468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198266    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\") pod \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") "
Oct 30 22:03:52.198468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198356    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\" (UniqueName: \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\") pod \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") "
Oct 30 22:03:52.198468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198435    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\") pod \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\" (UID: \"9beb9e4a-e922-4ae2-bccc-77039a16dd44\") "
Oct 30 22:03:52.198641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198503    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\") pod \"b68c5da5-3b11-4a61-899f-c518d2a3820f\" (UID: \"b68c5da5-3b11-4a61-899f-c518d2a3820f\") "
Oct 30 22:03:52.198745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.198716    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca" pod "9beb9e4a-e922-4ae2-bccc-77039a16dd44" (UID: "9beb9e4a-e922-4ae2-bccc-77039a16dd44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:52.199361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.199085    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca" pod "b68c5da5-3b11-4a61-899f-c518d2a3820f" (UID: "b68c5da5-3b11-4a61-899f-c518d2a3820f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:52.199493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.199389    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.199493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.199410    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.200597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.200570    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca" pod "9beb9e4a-e922-4ae2-bccc-77039a16dd44" (UID: "9beb9e4a-e922-4ae2-bccc-77039a16dd44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:52.200876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.200849    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj" pod "b68c5da5-3b11-4a61-899f-c518d2a3820f" (UID: "b68c5da5-3b11-4a61-899f-c518d2a3820f"). InnerVolumeSpecName "kube-api-access-nwqvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:52.200963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.200890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca" pod "b68c5da5-3b11-4a61-899f-c518d2a3820f" (UID: "b68c5da5-3b11-4a61-899f-c518d2a3820f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:52.200963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.200924    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q" pod "9beb9e4a-e922-4ae2-bccc-77039a16dd44" (UID: "9beb9e4a-e922-4ae2-bccc-77039a16dd44"). InnerVolumeSpecName "kube-api-access-kv69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:52.300413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.300381    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68c5da5-3b11-4a61-899f-c518d2a3820f-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.300413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.300412    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqvj\" (UniqueName: \"kubernetes.io/projected/b68c5da5-3b11-4a61-899f-c518d2a3820f-kube-api-access-nwqvj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.300610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.300424    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv69q\" (UniqueName: \"kubernetes.io/projected/9beb9e4a-e922-4ae2-bccc-77039a16dd44-kube-api-access-kv69q\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.300610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.300436    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9beb9e4a-e922-4ae2-bccc-77039a16dd44-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:52.735932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.735892    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:52.914623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.914589    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"
Oct 30 22:03:52.914623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.914609    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l" event={"ID":"9beb9e4a-e922-4ae2-bccc-77039a16dd44","Type":"ContainerDied","Data":"7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc"}
Oct 30 22:03:52.914914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.914661    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7798d675c365bb06d5fbcc600ba2aed477a3d809876cfce92465e36fbe2ff1cc"
Oct 30 22:03:52.916330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.916308    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"
Oct 30 22:03:52.916454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.916301    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh" event={"ID":"b68c5da5-3b11-4a61-899f-c518d2a3820f","Type":"ContainerDied","Data":"ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1"}
Oct 30 22:03:52.916454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:52.916445    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8a19b1e003fa779503c602237b65ef1cfc9e13079f08f13ec5cb3e6a5588c1"
Oct 30 22:03:54.017467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.017436    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:54.663830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.663791    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"]
Oct 30 22:03:54.664736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.664715    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b68c5da5-3b11-4a61-899f-c518d2a3820f" containerName="revoke-keys"
Oct 30 22:03:54.664816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.664739    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68c5da5-3b11-4a61-899f-c518d2a3820f" containerName="revoke-keys"
Oct 30 22:03:54.664866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.664848    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9beb9e4a-e922-4ae2-bccc-77039a16dd44" containerName="revoke-keys"
Oct 30 22:03:54.664866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.664858    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9beb9e4a-e922-4ae2-bccc-77039a16dd44" containerName="revoke-keys"
Oct 30 22:03:54.665036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.665023    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68c5da5-3b11-4a61-899f-c518d2a3820f" containerName="revoke-keys"
Oct 30 22:03:54.665094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.665052    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9beb9e4a-e922-4ae2-bccc-77039a16dd44" containerName="revoke-keys"
Oct 30 22:03:54.669226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.669203    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.711006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.710976    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:03:54.715185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.715163    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749271    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749319    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749348    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.749503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749408    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cf7q\" (UniqueName: \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.749503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749434    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749500    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749585    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42wn\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749615    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.749888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749648    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749766    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.749888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749817    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.750147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.749937    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.753186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.753157    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:03:54.753309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.753271    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:54.753373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.753334    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:54.753373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.753366    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:54.757498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.757474    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"]
Oct 30 22:03:54.774202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.774178    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:03:54.779199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.779177    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.803403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.803374    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:03:54.822281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.820317    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:03:54.850723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850692    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.850919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850734    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cf7q\" (UniqueName: \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.850919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850755    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.850919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850829    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.850919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850876    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.851165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.850955    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851038    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42wn\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851070    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.851165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851097    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.851165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851151    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851309    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.851461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851350    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.851461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851359    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851480    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851478    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851537    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.851615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851581    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851734    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851861    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.851907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.851894    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.852189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852163    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.852189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852184    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.852342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852218    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.852342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852249    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.852342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852303    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pk9k\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.852508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.852400    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.853138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.853113    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.854205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.853882    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.854205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.854026    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.854940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.854918    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.904803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.904698    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cf7q\" (UniqueName: \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\") pod \"maas-api-revoke-sub-4757509e-9nvhj\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:54.904803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.904725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.905388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.905357    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42wn\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:54.953677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953596    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.953677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953651    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.953923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953805    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.953923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953852    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pk9k\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.953923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953896    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953928    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.953954    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954103    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954148    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954213    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954421    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954721    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954801    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.954908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.954867    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.956182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.956164    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.956550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.956534    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:54.985704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:54.985675    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:03:55.007606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.007581    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:55.008215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.008193    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pk9k\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\") pod \"e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") " pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:55.029374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.029339    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:55.102254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.102209    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:55.373262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.373229    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"]
Oct 30 22:03:55.376725 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:55.376596    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebc0c7b_9ed5_4437_a8ca_c9f575848fc3.slice/crio-f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3 WatchSource:0}: Error finding container f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3: Status 404 returned error can't find the container with id f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3
Oct 30 22:03:55.407159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.407133    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:03:55.412857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.412835    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.448181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.448148    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:03:55.456487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.455980    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.456487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.456082    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.456487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.456120    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.477538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.477499    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgx2\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.477830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.477764    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.478025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478003    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.478162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478148    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.478357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478341    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.493200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478516    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.493200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478664    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.493200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.478817    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.493200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.479163    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.493200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.483876    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:03:55.514268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.514141    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:03:55.528736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.528684    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.528899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.528810    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.528899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.528846    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:55.580847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.580602    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.580847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.580694    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgx2\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.580847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.580715    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.580847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.580746    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.580847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.580764    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.581981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.581914    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.582351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.582289    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.582515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.582453    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.582739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.582701    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.582954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.582918    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.583802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.583756    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.584368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.584251    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.586010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.585051    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.586010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.585763    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.589602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.589561    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.590076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.590022    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.643839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.642767    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.648843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.648772    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgx2\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\") pod \"e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") " pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.727874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.727838    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:55.962172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.962127    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" event={"ID":"f73e9a15-8641-4194-8a92-bf5a8abf7a39","Type":"ContainerStarted","Data":"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"}
Oct 30 22:03:55.962362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.962202    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" event={"ID":"f73e9a15-8641-4194-8a92-bf5a8abf7a39","Type":"ContainerStarted","Data":"67cea27bc0934f859674d2807d691d0facfd76273ef6697c5be95a4642e022ef"}
Oct 30 22:03:55.968142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.968100    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" event={"ID":"11386dbe-d738-4a3d-9881-e20b55970f61","Type":"ContainerStarted","Data":"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"}
Oct 30 22:03:55.968304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.968168    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" event={"ID":"11386dbe-d738-4a3d-9881-e20b55970f61","Type":"ContainerStarted","Data":"a7ec7d7423c708bed0a915b09970a4ae80691c534662cc3946bb50130aec7d79"}
Oct 30 22:03:55.971094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.971066    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj" event={"ID":"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3","Type":"ContainerStarted","Data":"90b00fbd6ac970844fae06f4061b6224c3b9ccc092e4648c9a71decf88d55e00"}
Oct 30 22:03:55.971185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:55.971115    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj" event={"ID":"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3","Type":"ContainerStarted","Data":"f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3"}
Oct 30 22:03:56.029713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.029675    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:03:56.031352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.031325    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.031470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.031383    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:03:56.050465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.050427    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"]
Oct 30 22:03:56.065449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.065413    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b495d828-bhxwt"]
Oct 30 22:03:56.101806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.101065    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podStartSLOduration=2.101044715 podStartE2EDuration="2.101044715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.075160143 +0000 UTC m=+1245.059067019" watchObservedRunningTime="2026-10-30 22:03:56.101044715 +0000 UTC m=+1245.084951592"
Oct 30 22:03:56.107804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.104998    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:03:56.107804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.106071    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:03:56.107804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.107406    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.107804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.107466    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:03:56.113812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.113665    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.202853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.202676    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:03:56.220723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.217192    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:03:56.220723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.219236    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podStartSLOduration=2.219217023 podStartE2EDuration="2.219217023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.20754349 +0000 UTC m=+1245.191450367" watchObservedRunningTime="2026-10-30 22:03:56.219217023 +0000 UTC m=+1245.203123899"
Oct 30 22:03:56.240588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240556    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.240696 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240671    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:56.240769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240696    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.240769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240748    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:56.240940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240773    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.240940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240807    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:56.240940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240869    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.240940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240914    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.241144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240942    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.241144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.240975    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.241144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.241046    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.241144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.241086    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284cw\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342634    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342743    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342827    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342924    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.342954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342945    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.342967    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343015    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343040    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284cw\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343541    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343599    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343622    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.343898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.343862    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.344155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.344008    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.345623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.345602    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.346347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.346325    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.414019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.404961    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj" podStartSLOduration=2.404940078 podStartE2EDuration="2.404940078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.349230745 +0000 UTC m=+1245.333137621" watchObservedRunningTime="2026-10-30 22:03:56.404940078 +0000 UTC m=+1245.388846954"
Oct 30 22:03:56.417401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.417223    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284cw\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.418571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.418097    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\") pod \"e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") " pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:56.481856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:56.481563    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:57.027207 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:03:57.027015    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51537bc_dd14_43a0_964e_dae730ad5fb4.slice/crio-a4109a4b0710d97c49252d9b78fac7d08faf38f3baca4102b5a0423966ac6183 WatchSource:0}: Error finding container a4109a4b0710d97c49252d9b78fac7d08faf38f3baca4102b5a0423966ac6183: Status 404 returned error can't find the container with id a4109a4b0710d97c49252d9b78fac7d08faf38f3baca4102b5a0423966ac6183
Oct 30 22:03:57.028047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.028020    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:03:57.030445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.030412    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.030824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.030463    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:03:57.031761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.031716    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:57.031993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.031970    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:57.032168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.032142    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:03:57.046158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.046124    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" event={"ID":"e6dee765-db64-4ff6-981a-03feeda09b47","Type":"ContainerStarted","Data":"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"}
Oct 30 22:03:57.046689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.046658    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" event={"ID":"e6dee765-db64-4ff6-981a-03feeda09b47","Type":"ContainerStarted","Data":"96e9c08655b72b1ea766a73f6ead4fdab764c690803d96f788edf764c55aaca7"}
Oct 30 22:03:57.048828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.048806    5510 generic.go:356] "Generic (PLEG): container finished" podID="2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" containerID="90b00fbd6ac970844fae06f4061b6224c3b9ccc092e4648c9a71decf88d55e00" exitCode=0
Oct 30 22:03:57.049005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.048892    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj" event={"ID":"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3","Type":"ContainerDied","Data":"90b00fbd6ac970844fae06f4061b6224c3b9ccc092e4648c9a71decf88d55e00"}
Oct 30 22:03:57.106813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.106661    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.106813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.106719    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:03:57.140608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.140120    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548e4216-37f1-49c3-a1fe-666294e32e14" path="/var/lib/kubelet/pods/548e4216-37f1-49c3-a1fe-666294e32e14/volumes"
Oct 30 22:03:57.198837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.171210    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podStartSLOduration=2.171188871 podStartE2EDuration="2.171188871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:57.168288082 +0000 UTC m=+1246.152194960" watchObservedRunningTime="2026-10-30 22:03:57.171188871 +0000 UTC m=+1246.155095748"
Oct 30 22:03:57.732810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.732613    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:03:57.732810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.732729    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.732810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:57.732772    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:03:58.040959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.031298    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.040959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.031364    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:03:58.139807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.124656    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.139807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.124711    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:03:58.173474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.173434    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" event={"ID":"b51537bc-dd14-43a0-964e-dae730ad5fb4","Type":"ContainerStarted","Data":"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"}
Oct 30 22:03:58.173809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.173768    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" event={"ID":"b51537bc-dd14-43a0-964e-dae730ad5fb4","Type":"ContainerStarted","Data":"a4109a4b0710d97c49252d9b78fac7d08faf38f3baca4102b5a0423966ac6183"}
Oct 30 22:03:58.346860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.344357    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podStartSLOduration=2.344334319 podStartE2EDuration="2.344334319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:58.289989407 +0000 UTC m=+1247.273896284" watchObservedRunningTime="2026-10-30 22:03:58.344334319 +0000 UTC m=+1247.328241196"
Oct 30 22:03:58.391086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.391034    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.457234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.453861    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.490987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.490875    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:03:58.512714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.512482    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.512714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.512539    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:03:58.565188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.550491    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.565188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.550849    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:03:58.596822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.594330    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.628009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.627873    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.684806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.675920    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:03:58.734882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.734737    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.734882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.734815    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:03:58.736069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.735346    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.823999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.823955    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.866813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.863913    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:58.981802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:58.971919    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.082079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.033174    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.082079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.033230    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:03:59.082079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.075144    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:59.213202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.160842    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.707881    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.469432    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.467392    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.606102    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.708289    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.642906    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.686114    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.726503    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.726631    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.726669    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.726702    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.738998    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.739053    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.739139    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.775656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.740236    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:03:59.953455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.953398    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.960808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:03:59.954041    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.004805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.002952    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:00.004805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.003021    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:00.081854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.080755    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.081854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.080851    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:00.154897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.151015    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.154897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.151077    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:00.277813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.276607    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.312809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.311251    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:00.312809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.311326    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:00.450879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.353796    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:04:00.476812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.476445    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\") pod \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") "
Oct 30 22:04:00.476812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.476532    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\" (UniqueName: \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\") pod \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") "
Oct 30 22:04:00.476812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.476641    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\") pod \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\" (UID: \"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3\") "
Oct 30 22:04:00.493926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.485006    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca" pod "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" (UID: "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:00.548481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.548446    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.548792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.548747    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:00.674966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.574934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca" pod "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" (UID: "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:00.674966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.634448    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:00.674966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.634478    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:00.772656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.772390    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.810845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.796420    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.816577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.811118    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.816577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.813547    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.858319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.824868    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.858319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.833468    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 30 22:04:00.858319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.837046    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.858319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.837095    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:00.977818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:00.972508    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.003424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.002688    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.473152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.351393    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.473152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.439746    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.628833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.597396    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.628833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.597461    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.630704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q" pod "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" (UID: "2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3"). InnerVolumeSpecName "kube-api-access-2cf7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.637063    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cf7q\" (UniqueName: \"kubernetes.io/projected/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3-kube-api-access-2cf7q\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.638213    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.638261    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.638342    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.638388    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:01.671870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.666559    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.760879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.760031    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.760879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.760090    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:01.837823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.796890    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.837823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.797276    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.837823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.830257    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.873804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.838710    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.873804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:01.866866    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.192873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.082397    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.192873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.135543    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:02.192873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.135602    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:02.192873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.135705    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:02.192873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.135727    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:02.478986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.333658    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.706984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.706937    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.158716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.725558    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"
Oct 30 22:04:03.185389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.824921    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.185887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:02.825032    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.224325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.043519    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.231905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.085879    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.285835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.086057    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.311928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.304103    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:03.311928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.211234    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.311928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.304206    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:03.311928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.211319    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.311928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.304276    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:03.338511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.338294    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.400689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.398902    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.530677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.517291    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.530677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.517356    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:03.538400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.538340    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.545545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.545495    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.560292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.560100    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.825690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.561886    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.846819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.595945    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.846819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.832972    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:03.846819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.749454    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.846819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:03.833039    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:03.846819 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:03.611694    5510 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.335s"
Oct 30 22:04:04.032003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.031951    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.032341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.032311    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:04.122329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.107247    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.122329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.107306    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:04.122329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.110853    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.197671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.197627    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.279836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.242245    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.279836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.244975    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:04.422980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.383258    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"]
Oct 30 22:04:04.424441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.424410    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" containerName="revoke-keys"
Oct 30 22:04:04.424591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.424576    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" containerName="revoke-keys"
Oct 30 22:04:04.424864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.424848    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" containerName="revoke-keys"
Oct 30 22:04:04.427824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.399248    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.479194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.479161    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:04.498440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.482083    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:04.499610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.499583    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"]
Oct 30 22:04:04.524919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.524879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj" event={"ID":"2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3","Type":"ContainerDied","Data":"f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3"}
Oct 30 22:04:04.525198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.525176    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19f2f1dd91677db01434a3ad94e5b813e01c6a607e871a843ed33ff4a3327f3"
Oct 30 22:04:04.525362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.525343    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:04.525517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.525506    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:04.550512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.550467    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.563429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.495883    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.568055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.563065    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.601838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.600027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.601838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.600238    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnhg\" (UniqueName: \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.601838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.600314    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.620814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.603754    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.671294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.622260    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.671590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.671560    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:04.671724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.649224    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:04.671841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.649279    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:04.671921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.649723    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:04.672008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.644689    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:04.719117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.719092    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:04.719433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.719416    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"]
Oct 30 22:04:04.724846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.724819    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:04.736494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.736458    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.737078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.736731    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:04.790839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.751600    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.791424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.766301    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"]
Oct 30 22:04:04.792064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.792035    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.792329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.791925    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnhg\" (UniqueName: \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.792749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.792730    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.793367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.793264    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.800803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.793556    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:04.812813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.802830    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.887825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.868307    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnhg\" (UniqueName: \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\") pod \"maas-api-revoke-sub-ca202331-5gg4s\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.902813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.895924    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:04.923813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.918492    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9nb\" (UniqueName: \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:04.924827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.924752    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:04.928015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.924508    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:04.928516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:04.924124    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 22:04:05.030260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.030226    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.135851    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.124753    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.135942    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.135996    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.152385    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.152583    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.162804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.152643    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9nb\" (UniqueName: \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.261674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.261583    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.304418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.304382    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.325825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.325771    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:04:05.362219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.350171    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:05.456806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.431899    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:05.456806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.439970    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:04:05.456806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.440065    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:04:05.456806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.440123    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:05.457158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.457137    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:04:05.470445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.470411    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9nb\" (UniqueName: \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\") pod \"maas-api-revoke-sub-212c951d-r852w\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.541506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.541421    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.541803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.541741    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:05.686071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.686020    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:05.696438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.696394    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:05.707946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.707911    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:05.860521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.860408    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:05.876518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.876485    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.876823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.876772    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:05.988170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:05.988129    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:06.143529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.143406    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:06.453860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.445945    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:06.454498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.454462    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:06.519847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.500671    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:06.672864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.644200    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:06.699987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.673385    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:04:06.699987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.692716    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:04:06.725938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.711119    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:04:06.740831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.739934    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:04:06.745177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.741464    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:04:06.745177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.742690    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:06.749676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.748670    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:04:06.777267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.777216    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:04:06.777557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.777493    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:04:06.802607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.802529    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"]
Oct 30 22:04:06.851815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.851525    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:06.924317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.924102    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:06.941819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:06.941761    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:04:07.500293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.500254    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:07.500543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.500515    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 22:04:07.568371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.567512    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"]
Oct 30 22:04:07.737804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.735038    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:04:07.737804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.736605    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:04:07.738425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.737916    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:04:07.840467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.840380    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s" event={"ID":"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba","Type":"ContainerStarted","Data":"1651c9d74391fe9ae32abdd1db1c50463740278d1d3e759429d48cf89c4b0a88"}
Oct 30 22:04:07.840467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.840445    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s" event={"ID":"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba","Type":"ContainerStarted","Data":"6c30d71d64488b7829fc4787c62021c1c30d2f4a5506bd592cb6b89e2535ed7f"}
Oct 30 22:04:07.842907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.842837    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w" event={"ID":"f1e49b28-2047-4351-9f85-60a31a5d91ba","Type":"ContainerStarted","Data":"650a18ebb7c8e2d36a25d1c5a59e4a8b614f814b6fda961be180e2122ed4325b"}
Oct 30 22:04:07.988997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:07.988933    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s" podStartSLOduration=7.988912463 podStartE2EDuration="7.988912463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:07.982770232 +0000 UTC m=+1256.966677108" watchObservedRunningTime="2026-10-30 22:04:07.988912463 +0000 UTC m=+1256.972819341"
Oct 30 22:04:08.870408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.870367    5510 generic.go:356] "Generic (PLEG): container finished" podID="f1e49b28-2047-4351-9f85-60a31a5d91ba" containerID="347c431b3fa97df8064168f4accf36c10fc2029374ca5c15d8083fdbf5bf1cd7" exitCode=0
Oct 30 22:04:08.870906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.870702    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w" event={"ID":"f1e49b28-2047-4351-9f85-60a31a5d91ba","Type":"ContainerDied","Data":"347c431b3fa97df8064168f4accf36c10fc2029374ca5c15d8083fdbf5bf1cd7"}
Oct 30 22:04:08.878874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.878844    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:04:08.881486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.881466    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:04:08.882596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.882574    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:04:08.882761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.882741    5510 generic.go:356] "Generic (PLEG): container finished" podID="66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" containerID="1651c9d74391fe9ae32abdd1db1c50463740278d1d3e759429d48cf89c4b0a88" exitCode=0
Oct 30 22:04:08.883697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:08.883676    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s" event={"ID":"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba","Type":"ContainerDied","Data":"1651c9d74391fe9ae32abdd1db1c50463740278d1d3e759429d48cf89c4b0a88"}
Oct 30 22:04:08.942239 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:08.942188    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice/crio-conmon-347c431b3fa97df8064168f4accf36c10fc2029374ca5c15d8083fdbf5bf1cd7.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:09.255849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:09.254022    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:10.222896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.219003    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:10.241128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.239155    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.364481    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\" (UniqueName: \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\") pod \"f1e49b28-2047-4351-9f85-60a31a5d91ba\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.364678    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\") pod \"f1e49b28-2047-4351-9f85-60a31a5d91ba\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.364817    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\") pod \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.364867    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\") pod \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.364919    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\" (UniqueName: \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\") pod \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\" (UID: \"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.365003    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\") pod \"f1e49b28-2047-4351-9f85-60a31a5d91ba\" (UID: \"f1e49b28-2047-4351-9f85-60a31a5d91ba\") "
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.369152    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca" pod "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" (UID: "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.386012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.370497    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca" pod "f1e49b28-2047-4351-9f85-60a31a5d91ba" (UID: "f1e49b28-2047-4351-9f85-60a31a5d91ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.459136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.433987    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca" pod "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" (UID: "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:10.459136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.434196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb" pod "f1e49b28-2047-4351-9f85-60a31a5d91ba" (UID: "f1e49b28-2047-4351-9f85-60a31a5d91ba"). InnerVolumeSpecName "kube-api-access-sg9nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.516607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.516547    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9nb\" (UniqueName: \"kubernetes.io/projected/f1e49b28-2047-4351-9f85-60a31a5d91ba-kube-api-access-sg9nb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.516936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.516822    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.516936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.516873    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.516936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.516891    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.902805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.900031    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg" pod "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" (UID: "66874041-5ef0-4dbe-8ff2-fdace8f3f5ba"). InnerVolumeSpecName "kube-api-access-hdnhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.989016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:10.964557    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca" pod "f1e49b28-2047-4351-9f85-60a31a5d91ba" (UID: "f1e49b28-2047-4351-9f85-60a31a5d91ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:11.083851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.067541    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w" event={"ID":"f1e49b28-2047-4351-9f85-60a31a5d91ba","Type":"ContainerDied","Data":"650a18ebb7c8e2d36a25d1c5a59e4a8b614f814b6fda961be180e2122ed4325b"}
Oct 30 22:04:11.083851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.067665    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650a18ebb7c8e2d36a25d1c5a59e4a8b614f814b6fda961be180e2122ed4325b"
Oct 30 22:04:11.083851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.067798    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:04:11.083851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.072363    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnhg\" (UniqueName: \"kubernetes.io/projected/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba-kube-api-access-hdnhg\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.083851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.072397    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e49b28-2047-4351-9f85-60a31a5d91ba-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.504852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.504821    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.626992    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"]
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.628644    5510 cpu_manager.go:414] "containerMap: removing container" podUID="66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" containerName="revoke-keys"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.628666    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" containerName="revoke-keys"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.628732    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f1e49b28-2047-4351-9f85-60a31a5d91ba" containerName="revoke-keys"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.628741    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e49b28-2047-4351-9f85-60a31a5d91ba" containerName="revoke-keys"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.629119    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e49b28-2047-4351-9f85-60a31a5d91ba" containerName="revoke-keys"
Oct 30 22:04:11.629888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.629186    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" containerName="revoke-keys"
Oct 30 22:04:11.782856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.775096    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s" event={"ID":"66874041-5ef0-4dbe-8ff2-fdace8f3f5ba","Type":"ContainerDied","Data":"6c30d71d64488b7829fc4787c62021c1c30d2f4a5506bd592cb6b89e2535ed7f"}
Oct 30 22:04:11.782856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.775236    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c30d71d64488b7829fc4787c62021c1c30d2f4a5506bd592cb6b89e2535ed7f"
Oct 30 22:04:11.782856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.775295    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"]
Oct 30 22:04:11.803805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.797987    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:11.857805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.856846    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:11.873805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.870162    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:11.873805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.871000    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:11.910810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.902196    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kd8s\" (UniqueName: \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:11.911226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.911194    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:11.930805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.911614    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:11.977161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.977129    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:11.996003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.977691    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:11.996221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:11.977739    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:12.073857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.043843    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kd8s\" (UniqueName: \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.074250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.074217    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.074485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.074350    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:12.074722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.074696    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.075519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.075127    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.130807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.130730    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.161539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.161506    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kd8s\" (UniqueName: \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\") pod \"maas-api-revoke-sub-fef19fa3-cxpjc\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.161770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.161752    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"]
Oct 30 22:04:12.200035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.199998    5510 scope.go:122] "RemoveContainer" containerID="ed4ff87abad1afc10c5e577cf54919e429dafad95a7d483792a94a2b4539be6a"
Oct 30 22:04:12.200458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.200295    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.246887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.209306    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e56100e-gpwnv"]
Oct 30 22:04:12.259848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.259619    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:12.467957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.467909    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.469097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.468655    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.471454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.471432    5510 scope.go:122] "RemoveContainer" containerID="de9da4cd1f08b5b2786f96427b9be1d0a6a938d3a029b55239aceeb8cbadd05b"
Oct 30 22:04:12.502931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.502882    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.751850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.745971    5510 scope.go:122] "RemoveContainer" containerID="0bc28b50001d450f12214c53c8a1dec00a9a9d8251c66bc7c6adc487bf02890b"
Oct 30 22:04:12.810845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.799895    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:12.810845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.800636    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:04:12.863805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.861866    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.882391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.871953    5510 scope.go:122] "RemoveContainer" containerID="02dcbc31c4406a08be9e8269a6562f36fea7a78c55f9a921cbdb90e8b4243d47"
Oct 30 22:04:12.882391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:12.872545    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:04:13.072894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.072859    5510 scope.go:122] "RemoveContainer" containerID="83e6e0cd2bf60693483d7e0bd5fc8b8c757a7f2b2c5ec89d97934f640d1fe1b3"
Oct 30 22:04:13.165605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.165560    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.215842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.207320    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.242081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.242037    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.479334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.479290    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.487456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.487427    5510 scope.go:122] "RemoveContainer" containerID="5988b10784ae0d33300356745cbeab7aa981e63dce84fd5243785d96437001df"
Oct 30 22:04:13.488406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.488349    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.556976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.556931    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.559242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.559208    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705452ae-6e58-46ea-877b-2ac7d498071c" path="/var/lib/kubelet/pods/705452ae-6e58-46ea-877b-2ac7d498071c/volumes"
Oct 30 22:04:13.560002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.559982    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:13.560415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.560398    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:13.561522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.561506    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:13.576790 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.576736    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:13.806946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.806895    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.866894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.865487    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.974241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:13.960307    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.125867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.124822    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.281801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.262881    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.281801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.263621    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:14.281801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.273951    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.281801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.274577    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:14.367766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.282564    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.554943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.497916    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.554943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.498412    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.626804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.569554    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.360925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.360884    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.452264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.810965    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.454831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.327640    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.455029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.354919    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.455385 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:15.361870    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:04:15.455514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.368567    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.455851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.368602    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.476175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.407466    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.632856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.407519    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.657493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:14.769109    5510 scope.go:122] "RemoveContainer" containerID="1b5ed8649bfe050f3974fd8980ffca9e666b15279fe2cc24a781578e88ede259"
Oct 30 22:04:15.657851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.475549    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.658304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.499968    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.661801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.500550    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.716497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.716444    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:15.716829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.716767    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.889050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.863906    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:04:15.889050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.863981    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:04:15.889050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.867577    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:15.889050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:15.867649    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.367806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.367744    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.465168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.465127    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"]
Oct 30 22:04:16.508740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.508594    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.584951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.584892    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.632270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.632146    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.643190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.643138    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.690822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.683328    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.811814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.692935    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.816419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.722708    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.817146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.753377    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 22:04:16.817308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.797739    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.925702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:16.925562    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.082806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.057195    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.082806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.057307    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.413886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.413825    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.597116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.593262    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.622825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.621963    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.647806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.637501    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.648085    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.906347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.768972    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.906640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.906588    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.906864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.853159    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.907500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.853123    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.907854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.852992    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.908038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.908014    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.908290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.904449    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.908290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.908215    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.908604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.853184    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.908942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.853217    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.925650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.925518    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:17.932819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:17.932762    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.496950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.409452    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc" event={"ID":"7171a3bf-89f7-4b20-a837-f1caa7d08c23","Type":"ContainerStarted","Data":"858be7382e455ae4d22e9ed4fcaeacbfb48775dc578d67ded02242db28382f7f"}
Oct 30 22:04:18.517827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.455754    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.543815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.509055    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.677967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.538409    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:18.678276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.678241    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.678406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.631660    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.678525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.631699    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:18.679111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.679073    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.631596    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.538439    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.724525    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.670977    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.724601    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.740013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.631734    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.942707    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.943238    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.943576    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.947893    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.955214    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.955294    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:18.955575    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:19.151570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.151531    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:19.551812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.539866    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.223868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.704531    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"]
Oct 30 22:04:20.245832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.706227    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Oct 30 22:04:20.270807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.726677    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.726714    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.726736    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.915933    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.959486    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.352918    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.098162    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:19.974508    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.141508    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.355820    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.129323    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.356545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.098241    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.357311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.129269    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.358454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.098204    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:20.520863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.513237    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:20.520863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.513336    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:20.533644 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:20.533602    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:20.550669 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:20.550610    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:20.567023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.566977    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.716857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.711878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77h68\" (UniqueName: \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.716857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.711988    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.716857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.712033    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.716857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.712768    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 30 22:04:20.773890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.773244    5510 generic.go:356] "Generic (PLEG): container finished" podID="7171a3bf-89f7-4b20-a837-f1caa7d08c23" containerID="6c21b15c90caf39e852fa6e990ef27560e59c152ce153be7a503641334500a37" exitCode=0
Oct 30 22:04:20.842107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.842057    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77h68\" (UniqueName: \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.842468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.842444    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.842648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.842631    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.844452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.844407    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:20.932298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:20.932259    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:21.025319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.001510    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77h68\" (UniqueName: \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\") pod \"maas-api-revoke-sub-1a4b2d0a-r9jkq\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:21.165813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.152515    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.291124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.228571    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:21.339230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.339183    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.354854    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.354948    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.354997    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.355048    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.364978    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.365106    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.365190    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:21.366869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.365244    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:21.418160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.418024    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"]
Oct 30 22:04:21.822057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.821998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc" event={"ID":"7171a3bf-89f7-4b20-a837-f1caa7d08c23","Type":"ContainerDied","Data":"6c21b15c90caf39e852fa6e990ef27560e59c152ce153be7a503641334500a37"}
Oct 30 22:04:21.895835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.875309    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:21.895835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:21.875384    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:22.222017 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:22.221966    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:22.332861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.311231    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:22.356699 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:22.356657    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:22.419175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.419128    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.442849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.442706    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.452078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.452038    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:22.477547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.454371    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.484473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.454408    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.505893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.454444    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.664760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.454465    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.665331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.454497    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.665683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.618417    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:22.677125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.670005    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.796042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:22.796001    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.043278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.043239    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:23.074925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.065306    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:04:23.074925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.065366    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:23.125926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.102642    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:04:23.125926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.105533    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:23.141825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.129280    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:04:23.141825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.141635    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:04:23.183990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.183957    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:23.192358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.192329    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:23.194919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.194887    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:23.196976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.196890    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:23.199461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.199373    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"
Oct 30 22:04:23.199461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.199384    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc" event={"ID":"7171a3bf-89f7-4b20-a837-f1caa7d08c23","Type":"ContainerDied","Data":"858be7382e455ae4d22e9ed4fcaeacbfb48775dc578d67ded02242db28382f7f"}
Oct 30 22:04:23.199461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.199432    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858be7382e455ae4d22e9ed4fcaeacbfb48775dc578d67ded02242db28382f7f"
Oct 30 22:04:23.221491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.221462    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"]
Oct 30 22:04:23.226147 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:23.226110    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6013d903_973f_40ca_b72a_f6affaf888bc.slice/crio-22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92 WatchSource:0}: Error finding container 22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92: Status 404 returned error can't find the container with id 22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92
Oct 30 22:04:23.356618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.356532    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\") pod \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") "
Oct 30 22:04:23.356763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.356735    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\" (UniqueName: \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\") pod \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") "
Oct 30 22:04:23.356839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.356822    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\") pod \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\" (UID: \"7171a3bf-89f7-4b20-a837-f1caa7d08c23\") "
Oct 30 22:04:23.358332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.358294    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca" pod "7171a3bf-89f7-4b20-a837-f1caa7d08c23" (UID: "7171a3bf-89f7-4b20-a837-f1caa7d08c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:23.452890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.452849    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca" pod "7171a3bf-89f7-4b20-a837-f1caa7d08c23" (UID: "7171a3bf-89f7-4b20-a837-f1caa7d08c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:23.452890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.452874    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s" pod "7171a3bf-89f7-4b20-a837-f1caa7d08c23" (UID: "7171a3bf-89f7-4b20-a837-f1caa7d08c23"). InnerVolumeSpecName "kube-api-access-4kd8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:23.459746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.459711    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:23.459929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.459750    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171a3bf-89f7-4b20-a837-f1caa7d08c23-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:23.459929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.459766    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kd8s\" (UniqueName: \"kubernetes.io/projected/7171a3bf-89f7-4b20-a837-f1caa7d08c23-kube-api-access-4kd8s\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:23.882050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:23.881891    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:24.222305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:24.222269    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq" event={"ID":"6013d903-973f-40ca-b72a-f6affaf888bc","Type":"ContainerStarted","Data":"b9e8f4ff56302f69af693278748dedd08b71387f76607c59a3965c95c8af669c"}
Oct 30 22:04:24.222905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:24.222880    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq" event={"ID":"6013d903-973f-40ca-b72a-f6affaf888bc","Type":"ContainerStarted","Data":"22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92"}
Oct 30 22:04:24.403138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:24.403062    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq" podStartSLOduration=6.403043739 podStartE2EDuration="6.403043739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:24.400338344 +0000 UTC m=+1273.384245222" watchObservedRunningTime="2026-10-30 22:04:24.403043739 +0000 UTC m=+1273.386950616"
Oct 30 22:04:25.231272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:25.231233    5510 generic.go:356] "Generic (PLEG): container finished" podID="6013d903-973f-40ca-b72a-f6affaf888bc" containerID="b9e8f4ff56302f69af693278748dedd08b71387f76607c59a3965c95c8af669c" exitCode=0
Oct 30 22:04:25.231678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:25.231312    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq" event={"ID":"6013d903-973f-40ca-b72a-f6affaf888bc","Type":"ContainerDied","Data":"b9e8f4ff56302f69af693278748dedd08b71387f76607c59a3965c95c8af669c"}
Oct 30 22:04:26.412850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.412820    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:26.482550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.482513    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\") pod \"6013d903-973f-40ca-b72a-f6affaf888bc\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") "
Oct 30 22:04:26.482745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.482558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\" (UniqueName: \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\") pod \"6013d903-973f-40ca-b72a-f6affaf888bc\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") "
Oct 30 22:04:26.482745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.482626    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\") pod \"6013d903-973f-40ca-b72a-f6affaf888bc\" (UID: \"6013d903-973f-40ca-b72a-f6affaf888bc\") "
Oct 30 22:04:26.483006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.482982    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca" pod "6013d903-973f-40ca-b72a-f6affaf888bc" (UID: "6013d903-973f-40ca-b72a-f6affaf888bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:26.483186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.483168    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:26.484658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.484625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68" pod "6013d903-973f-40ca-b72a-f6affaf888bc" (UID: "6013d903-973f-40ca-b72a-f6affaf888bc"). InnerVolumeSpecName "kube-api-access-77h68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:26.484769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.484712    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca" pod "6013d903-973f-40ca-b72a-f6affaf888bc" (UID: "6013d903-973f-40ca-b72a-f6affaf888bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:26.584220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.584186    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6013d903-973f-40ca-b72a-f6affaf888bc-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:26.584220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:26.584215    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77h68\" (UniqueName: \"kubernetes.io/projected/6013d903-973f-40ca-b72a-f6affaf888bc-kube-api-access-77h68\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:27.252958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:27.252931    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"
Oct 30 22:04:27.253126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:27.252935    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq" event={"ID":"6013d903-973f-40ca-b72a-f6affaf888bc","Type":"ContainerDied","Data":"22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92"}
Oct 30 22:04:27.253126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:27.253053    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22791b2058fe2b390bb1938cf5dffb58f6cdc15295bd9cf019a0a1fac76ada92"
Oct 30 22:04:27.721078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:27.721050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:30.810255 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:30.810219    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:30.865408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.865374    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"]
Oct 30 22:04:30.866355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866325    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6013d903-973f-40ca-b72a-f6affaf888bc" containerName="revoke-keys"
Oct 30 22:04:30.866355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866354    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6013d903-973f-40ca-b72a-f6affaf888bc" containerName="revoke-keys"
Oct 30 22:04:30.866550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866371    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7171a3bf-89f7-4b20-a837-f1caa7d08c23" containerName="revoke-keys"
Oct 30 22:04:30.866550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866381    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171a3bf-89f7-4b20-a837-f1caa7d08c23" containerName="revoke-keys"
Oct 30 22:04:30.866656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866600    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6013d903-973f-40ca-b72a-f6affaf888bc" containerName="revoke-keys"
Oct 30 22:04:30.866656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.866615    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171a3bf-89f7-4b20-a837-f1caa7d08c23" containerName="revoke-keys"
Oct 30 22:04:30.870641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.870620    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:30.922726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.922692    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"]
Oct 30 22:04:30.956481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.956434    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg9t\" (UniqueName: \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:30.956689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.956500    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:30.956689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.956602    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:30.963281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.963248    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:30.963281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.963250    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:30.963531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.963320    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:30.965521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:30.965496    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:30.982052 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:30.982011    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:31.057997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.057953    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg9t\" (UniqueName: \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.058186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.058051    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.058186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.058088    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.058896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.058866    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.060549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.060490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.111933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.111898    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg9t\" (UniqueName: \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\") pod \"maas-api-revoke-sub-eee74d0b-v5htr\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.185567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.185535    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:31.566878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:31.566846    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"]
Oct 30 22:04:31.569284 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:31.569236    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e46a775_ff97_4343_a067_b16e1a884f93.slice/crio-8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c WatchSource:0}: Error finding container 8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c: Status 404 returned error can't find the container with id 8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c
Oct 30 22:04:32.142555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:32.142473    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:32.333125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:32.333089    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr" event={"ID":"7e46a775-ff97-4343-a067-b16e1a884f93","Type":"ContainerStarted","Data":"4ece35a76bb507b7986e5ebe3327c39fbf91cd58d17832e59c502469fb854316"}
Oct 30 22:04:32.333404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:32.333378    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr" event={"ID":"7e46a775-ff97-4343-a067-b16e1a884f93","Type":"ContainerStarted","Data":"8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c"}
Oct 30 22:04:33.074409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.074344    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr" podStartSLOduration=3.074321436 podStartE2EDuration="3.074321436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:32.440926633 +0000 UTC m=+1281.424833514" watchObservedRunningTime="2026-10-30 22:04:33.074321436 +0000 UTC m=+1282.058228314"
Oct 30 22:04:33.079392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.079363    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"]
Oct 30 22:04:33.083804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.083772    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.157307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.157269    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"]
Oct 30 22:04:33.222168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.222132    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzrs\" (UniqueName: \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.222339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.222274    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.222441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.222426    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.323832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.323759    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.324011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.323905    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzrs\" (UniqueName: \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.324011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.323989    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.324482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.324422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.326310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.326289    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.342754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.342721    5510 generic.go:356] "Generic (PLEG): container finished" podID="7e46a775-ff97-4343-a067-b16e1a884f93" containerID="4ece35a76bb507b7986e5ebe3327c39fbf91cd58d17832e59c502469fb854316" exitCode=0
Oct 30 22:04:33.342890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.342809    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr" event={"ID":"7e46a775-ff97-4343-a067-b16e1a884f93","Type":"ContainerDied","Data":"4ece35a76bb507b7986e5ebe3327c39fbf91cd58d17832e59c502469fb854316"}
Oct 30 22:04:33.378150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.378113    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzrs\" (UniqueName: \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\") pod \"maas-api-revoke-sub-1f197684-vdxnc\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.407152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.407118    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:33.794911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:33.794867    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"]
Oct 30 22:04:33.800774 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:33.800743    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988aff3b_a219_4a4e_b210_c2aa6e4768fb.slice/crio-76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245 WatchSource:0}: Error finding container 76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245: Status 404 returned error can't find the container with id 76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245
Oct 30 22:04:34.352952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.352909    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc" event={"ID":"988aff3b-a219-4a4e-b210-c2aa6e4768fb","Type":"ContainerStarted","Data":"fc54d02be1ac9992f1fb326633bc76db2a8db40a34c4f8d74f488afc4aa22983"}
Oct 30 22:04:34.353465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.352970    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc" event={"ID":"988aff3b-a219-4a4e-b210-c2aa6e4768fb","Type":"ContainerStarted","Data":"76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245"}
Oct 30 22:04:34.463956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.463896    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc" podStartSLOduration=1.4638779149999999 podStartE2EDuration="1.463877915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:34.460939748 +0000 UTC m=+1283.444846636" watchObservedRunningTime="2026-10-30 22:04:34.463877915 +0000 UTC m=+1283.447784794"
Oct 30 22:04:34.548013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.547981    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:34.644807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.644756    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\" (UniqueName: \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\") pod \"7e46a775-ff97-4343-a067-b16e1a884f93\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") "
Oct 30 22:04:34.644988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.644974    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\") pod \"7e46a775-ff97-4343-a067-b16e1a884f93\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") "
Oct 30 22:04:34.645085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.645074    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\") pod \"7e46a775-ff97-4343-a067-b16e1a884f93\" (UID: \"7e46a775-ff97-4343-a067-b16e1a884f93\") "
Oct 30 22:04:34.645473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.645443    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca" pod "7e46a775-ff97-4343-a067-b16e1a884f93" (UID: "7e46a775-ff97-4343-a067-b16e1a884f93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:34.645627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.645611    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:34.646869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.646839    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca" pod "7e46a775-ff97-4343-a067-b16e1a884f93" (UID: "7e46a775-ff97-4343-a067-b16e1a884f93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:34.646970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.646956    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t" pod "7e46a775-ff97-4343-a067-b16e1a884f93" (UID: "7e46a775-ff97-4343-a067-b16e1a884f93"). InnerVolumeSpecName "kube-api-access-5vg9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:34.747039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.747007    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e46a775-ff97-4343-a067-b16e1a884f93-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:34.747039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:34.747036    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg9t\" (UniqueName: \"kubernetes.io/projected/7e46a775-ff97-4343-a067-b16e1a884f93-kube-api-access-5vg9t\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:35.362359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:35.362327    5510 generic.go:356] "Generic (PLEG): container finished" podID="988aff3b-a219-4a4e-b210-c2aa6e4768fb" containerID="fc54d02be1ac9992f1fb326633bc76db2a8db40a34c4f8d74f488afc4aa22983" exitCode=0
Oct 30 22:04:35.362881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:35.362417    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc" event={"ID":"988aff3b-a219-4a4e-b210-c2aa6e4768fb","Type":"ContainerDied","Data":"fc54d02be1ac9992f1fb326633bc76db2a8db40a34c4f8d74f488afc4aa22983"}
Oct 30 22:04:35.363911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:35.363894    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"
Oct 30 22:04:35.364053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:35.363891    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr" event={"ID":"7e46a775-ff97-4343-a067-b16e1a884f93","Type":"ContainerDied","Data":"8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c"}
Oct 30 22:04:35.364053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:35.364005    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ceb13bbdedbec22c1a97af137730b4fe24720268ed785c906cfac488cbb899c"
Oct 30 22:04:36.540411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.540383    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:36.541682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.541659    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:36.702850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.701540    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\" (UniqueName: \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\") pod \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") "
Oct 30 22:04:36.702850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.701599    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\") pod \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") "
Oct 30 22:04:36.702850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.701761    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\") pod \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\" (UID: \"988aff3b-a219-4a4e-b210-c2aa6e4768fb\") "
Oct 30 22:04:36.704871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.704151    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca" pod "988aff3b-a219-4a4e-b210-c2aa6e4768fb" (UID: "988aff3b-a219-4a4e-b210-c2aa6e4768fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:36.705320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.705292    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs" pod "988aff3b-a219-4a4e-b210-c2aa6e4768fb" (UID: "988aff3b-a219-4a4e-b210-c2aa6e4768fb"). InnerVolumeSpecName "kube-api-access-hbzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:36.707007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.706988    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca" pod "988aff3b-a219-4a4e-b210-c2aa6e4768fb" (UID: "988aff3b-a219-4a4e-b210-c2aa6e4768fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:36.804017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.803976    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzrs\" (UniqueName: \"kubernetes.io/projected/988aff3b-a219-4a4e-b210-c2aa6e4768fb-kube-api-access-hbzrs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:36.804017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.804020    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:36.804223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:36.804037    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/988aff3b-a219-4a4e-b210-c2aa6e4768fb-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:37.383840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:37.383775    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"
Oct 30 22:04:37.383840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:37.383816    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc" event={"ID":"988aff3b-a219-4a4e-b210-c2aa6e4768fb","Type":"ContainerDied","Data":"76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245"}
Oct 30 22:04:37.384064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:37.383868    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a2aa707cd4e03b55a8a6c9e1efeb7fc6e3ab90145a57519794d59eb0e5a245"
Oct 30 22:04:37.769920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:37.769885    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:37.829576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:37.829545    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:04:38.695810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:38.689573    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:38.868287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:38.868238    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:39.450637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.450600    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:04:39.453606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.453577    5510 cpu_manager.go:414] "containerMap: removing container" podUID="988aff3b-a219-4a4e-b210-c2aa6e4768fb" containerName="revoke-keys"
Oct 30 22:04:39.453856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.453836    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988aff3b-a219-4a4e-b210-c2aa6e4768fb" containerName="revoke-keys"
Oct 30 22:04:39.454027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.454012    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7e46a775-ff97-4343-a067-b16e1a884f93" containerName="revoke-keys"
Oct 30 22:04:39.454117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.454106    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e46a775-ff97-4343-a067-b16e1a884f93" containerName="revoke-keys"
Oct 30 22:04:39.454360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.454346    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="988aff3b-a219-4a4e-b210-c2aa6e4768fb" containerName="revoke-keys"
Oct 30 22:04:39.454452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.454442    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e46a775-ff97-4343-a067-b16e1a884f93" containerName="revoke-keys"
Oct 30 22:04:39.466444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.465284    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.486803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.479063    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.655830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.652237    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.686802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.673738    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.710989    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711065    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711164    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdmjs\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711196    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711401    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.711552    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.715892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.678053    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.748975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.748802    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:04:39.835141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.835100    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.846943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.846899    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.847258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.847232    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdmjs\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.847411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.847393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.847654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.847637    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.847817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.847802    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.847972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.847960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.848113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.848100    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.848243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.848231    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.849142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.849118    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.849822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.849800    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.851619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.851598    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.871845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.859347    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.872406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.860332    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.875052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.875018    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.875190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.875091    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.937517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.937471    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.939306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.939263    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:39.998855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:39.994705    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdmjs\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:40.016813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.013446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:40.047347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.047312    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\") pod \"e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") " pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:40.095834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.091152    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:40.394939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.394888    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.444844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.436902    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.480821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.466523    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.481518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.481484    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:40.674858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.672120    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:40.674858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.673456    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:40.674858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.674554    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:40.692908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.692862    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 30 22:04:40.729810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.720582    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:40.736311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.736279    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:04:40.993916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.993581    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.994400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:40.994233    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.009807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.000685    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.048769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.039304    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:04:41.049253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.049231    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:04:41.051936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.047965    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.207965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.207928    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:41.706599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.706498    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.745843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.706961    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.796810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.771562    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.031022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.739313    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.031022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:41.709656    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.031022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.015499    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.188861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.061258    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:42.188861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.061395    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:42.188861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.061432    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:42.188861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.061481    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.188861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.142137    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.285857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.194869    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.344831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.222989    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.344831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.311964    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:04:42.777151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.759760    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.777151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.760064    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.921822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.921123    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.967805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.966997    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.984807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.966950    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.984807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:42.982915    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.034869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.034711    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.070946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.070899    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:43.155632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.155578    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:43.308156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.308048    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:43.359843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.350127    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:43.359843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.350234    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:43.359843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.350308    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:04:43.359843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.334063    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.791807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.772170    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.792358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.772690    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.933820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.891109    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:44.035860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.965213    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:04:44.040086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.040053    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" event={"ID":"02097807-fb39-45aa-8500-26b1afe0175c","Type":"ContainerStarted","Data":"d38c590c4c9959915749940d4ecd81ca7c1075a2fb5c87442d3c91428cb2cc7a"}
Oct 30 22:04:44.076262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.010318    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.083899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.010361    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.087960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.039918    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.097830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:43.987882    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.098370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.075869    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.229806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.223984    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.258801 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:44.230482    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:04:44.501533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.501489    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:44.658932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.658880    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.749806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.724065    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:44.832890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.784177    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"]
Oct 30 22:04:44.912149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.912089    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.934952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.924729    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:44.997391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:44.997361    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:45.088183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.088056    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.198655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.100512    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:45.205732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.100816    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:45.210182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.101020    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.294933    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.120554    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.120771    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.127478    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.127715    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.305826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.167886    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.425545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.425431    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t8b8g" podUID="4f613d67-925e-409b-9208-13ecbb1163a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 22:04:45.426554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.426520    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.101219    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.804499    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.873414    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.901222    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg62j\" (UniqueName: \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.046416    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.056840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.046667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.080017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.941869    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:46.081423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.081376    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.131283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:45.942715    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:46.131898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.131845    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.132132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.081712    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.371409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.371309    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:46.629856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.598316    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.730801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.724049    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.730801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.604927    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.730801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.714031    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h" podUID="a0d8ee34-d09b-4cc9-a870-78846d79306c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.730801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.714668    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 22:04:46.731192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.730992    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr" podUID="7cafbb56-06c8-47ab-9ec6-d5355209b352" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.731673 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.604878    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.764852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.754017    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg62j\" (UniqueName: \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.764852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.754117    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.764852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.754310    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:46.844549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.844508    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:46.844840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.844806    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:46.845963 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:46.845933    5510 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.539s"
Oct 30 22:04:46.846269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.846250    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"]
Oct 30 22:04:46.984868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.882624    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:46.986560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:46.986522    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:47.068807 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:47.034717    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:47.102803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.065640    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:47.142038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.139421    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:47.167974 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:47.148624    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:47.213818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.197721    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:47.235885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.222340    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:47.235885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.232918    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:47.272806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.236172    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:47.321748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.321713    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:47.740810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.736423    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:47.765805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.764079    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:47.842458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.842386    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg62j\" (UniqueName: \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:47.972806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.968891    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:48.030982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.030875    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.031238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.980923    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:48.032158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.980885    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.033524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.033490    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:48.034440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.034411    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.062635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:47.981697    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151b9b47-jbrjp\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:48.093703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.093663    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:48.101829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.093937    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.114900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.114863    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:48.118893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.118849    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.186855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.150707    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:04:48.194949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.179241    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"]
Oct 30 22:04:48.226028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.225885    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.266861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.253279    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.298125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.253336    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:48.343466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.310867    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"]
Oct 30 22:04:48.366314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.312064    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.378755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.378720    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:48.398577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.398539    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.424509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.424477    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:48.461805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.432268    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.467629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.467593    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.497805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.472272    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdwm\" (UniqueName: \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.613803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.508198    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.643810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.618310    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.755030    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.774599    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.797752    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.797827    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.825062    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdwm\" (UniqueName: \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.825207    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.825234    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.881911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.834528    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.963280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.963245    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:48.979857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:48.979167    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.123501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.123459    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.125039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.125001    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.184875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.154410    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdwm\" (UniqueName: \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\") pod \"maas-api-revoke-sub-09941488-bzw9r\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:49.331949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.331885    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.384807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.383878    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.385760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.385717    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.423807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.392345    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:49.423807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.392407    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.426759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.426717    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.454859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.454562    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.454859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.454805    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:49.527848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.487328    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.562830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.539647    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:49.628276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.628231    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:49.642804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.630811    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Oct 30 22:04:49.732809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.716122    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.759526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.759479    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:04:49.762930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.762894    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:49.763097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.762952    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:49.793806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.770467    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podStartSLOduration=10.770448043 podStartE2EDuration="10.770448043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:49.600556921 +0000 UTC m=+1298.584463797" watchObservedRunningTime="2026-10-30 22:04:49.770448043 +0000 UTC m=+1298.754354919"
Oct 30 22:04:49.831803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:49.799029    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:04:50.142742 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:50.142648    5510 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.015s"
Oct 30 22:04:50.158806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.142979    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" event={"ID":"02097807-fb39-45aa-8500-26b1afe0175c","Type":"ContainerStarted","Data":"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"}
Oct 30 22:04:50.162266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.149679    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:50.165910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.165867    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:50.166116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.149770    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:04:50.176384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.169923    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:50.176384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.149727    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:04:50.395854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.374477    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:50.397552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.397495    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.428925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.428874    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.458885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.458842    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.494804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.494418    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.667397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.642089    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.717691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.717651    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.718102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.717980    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.762017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.761955    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:50.924864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:50.923201    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:04:51.028953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.028919    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:04:51.030670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.030641    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:04:51.085800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.031512    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:04:51.085800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.031561    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:04:51.103436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.103311    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:04:51.106501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.106466    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:51.106632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.106523    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:51.127012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.115715    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:04:51.127012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.116742    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:04:51.205493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.205381    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:04:51.209315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.208497    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:04:51.240669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.240640    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:51.398681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.398644    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"]
Oct 30 22:04:51.418795 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:51.418730    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12959834_6b5c_4fef_9d48_c58f4dec0314.slice/crio-6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65 WatchSource:0}: Error finding container 6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65: Status 404 returned error can't find the container with id 6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65
Oct 30 22:04:51.419588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.419369    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"]
Oct 30 22:04:51.448583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.448554    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:51.634520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.634484    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:04:51.640994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.640968    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp" event={"ID":"12959834-6b5c-4fef-9d48-c58f4dec0314","Type":"ContainerStarted","Data":"6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65"}
Oct 30 22:04:51.641159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.641060    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:51.642451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.642356    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r" event={"ID":"8bf16d92-b028-48ba-883c-c545d3dc64bd","Type":"ContainerStarted","Data":"c05af6b42f711f4655467a32b86b5312593dea4e0f9b0e5689afcad0c13bbacd"}
Oct 30 22:04:51.689239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.689206    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b3084d7f\""
Oct 30 22:04:51.691398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.691366    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:51.691771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.691744    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b3084d7f-dockercfg-wnj84\""
Oct 30 22:04:51.698738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.698713    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:04:51.811857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.796570    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:51.822867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.812326    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpdd\" (UniqueName: \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.008811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.973917    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.008811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.974166    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpdd\" (UniqueName: \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.008811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:51.983132    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:04:52.024114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.010292    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.119849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.099875    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpdd\" (UniqueName: \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\") pod \"payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.187800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.182164    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:52.187800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.182225    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:52.221892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.205600    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.344361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.344329    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:52.391856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.352066    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.391856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.352147    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlbm\" (UniqueName: \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.444204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.437756    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:04:52.512310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.512272    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.516716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.516488    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.516931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.516720    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlbm\" (UniqueName: \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.602058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.602003    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlbm\" (UniqueName: \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\") pod \"payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") " pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.696906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.696867    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp" event={"ID":"12959834-6b5c-4fef-9d48-c58f4dec0314","Type":"ContainerStarted","Data":"c9d6a7eeac1ae41b46b5a72e5d35f6c1f833640825c9e154699cc58cc48e7785"}
Oct 30 22:04:52.700223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.700185    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r" event={"ID":"8bf16d92-b028-48ba-883c-c545d3dc64bd","Type":"ContainerStarted","Data":"7c17bf86c7dd3ea871814c18aaee4eefff58b46d96dff0d4e84c8490c38abf8f"}
Oct 30 22:04:52.801203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.801167    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:52.805300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:52.805249    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp" podStartSLOduration=10.805233719 podStartE2EDuration="10.805233719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:52.804141682 +0000 UTC m=+1301.788048558" watchObservedRunningTime="2026-10-30 22:04:52.805233719 +0000 UTC m=+1301.789140597"
Oct 30 22:04:53.005097 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:53.005057    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986cc2c6_8495_46d8_9e40_9227ba115da3.slice/crio-16267f954871b762543e648af7ed3a9dd423653206dfe76485cba950b9f39efc WatchSource:0}: Error finding container 16267f954871b762543e648af7ed3a9dd423653206dfe76485cba950b9f39efc: Status 404 returned error can't find the container with id 16267f954871b762543e648af7ed3a9dd423653206dfe76485cba950b9f39efc
Oct 30 22:04:53.010561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.008695    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r" podStartSLOduration=9.008680175 podStartE2EDuration="9.008680175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:52.910685915 +0000 UTC m=+1301.894592793" watchObservedRunningTime="2026-10-30 22:04:53.008680175 +0000 UTC m=+1301.992587051"
Oct 30 22:04:53.010561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.008938    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:04:53.093962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.093933    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:53.094110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.094020    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:53.219880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.219848    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:04:53.713373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.713337    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" event={"ID":"9b210c05-2ead-4555-9b40-e55b0aa3be59","Type":"ContainerStarted","Data":"d7f82528ed77d8b3d1e57023832274c9800e58a33769ab26219a05d4586f7b9e"}
Oct 30 22:04:53.713869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.713403    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" event={"ID":"9b210c05-2ead-4555-9b40-e55b0aa3be59","Type":"ContainerStarted","Data":"90351a27adb868d2e0bcf195fd6d61852467223c0c4f3d5efbf7b40cc5b8939b"}
Oct 30 22:04:53.713869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.713816    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:04:53.715320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.715293    5510 generic.go:356] "Generic (PLEG): container finished" podID="12959834-6b5c-4fef-9d48-c58f4dec0314" containerID="c9d6a7eeac1ae41b46b5a72e5d35f6c1f833640825c9e154699cc58cc48e7785" exitCode=0
Oct 30 22:04:53.715418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.715373    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp" event={"ID":"12959834-6b5c-4fef-9d48-c58f4dec0314","Type":"ContainerDied","Data":"c9d6a7eeac1ae41b46b5a72e5d35f6c1f833640825c9e154699cc58cc48e7785"}
Oct 30 22:04:53.717104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.717083    5510 generic.go:356] "Generic (PLEG): container finished" podID="8bf16d92-b028-48ba-883c-c545d3dc64bd" containerID="7c17bf86c7dd3ea871814c18aaee4eefff58b46d96dff0d4e84c8490c38abf8f" exitCode=0
Oct 30 22:04:53.717197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.717157    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r" event={"ID":"8bf16d92-b028-48ba-883c-c545d3dc64bd","Type":"ContainerDied","Data":"7c17bf86c7dd3ea871814c18aaee4eefff58b46d96dff0d4e84c8490c38abf8f"}
Oct 30 22:04:53.718898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.718875    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" event={"ID":"986cc2c6-8495-46d8-9e40-9227ba115da3","Type":"ContainerStarted","Data":"1da132428adaf7aeabfe2071aefdc5a210de32c9832c0260f6273f1a89090a71"}
Oct 30 22:04:53.718987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.718921    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:04:53.718987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.718943    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" event={"ID":"986cc2c6-8495-46d8-9e40-9227ba115da3","Type":"ContainerStarted","Data":"16267f954871b762543e648af7ed3a9dd423653206dfe76485cba950b9f39efc"}
Oct 30 22:04:53.831527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:53.831473    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" podStartSLOduration=2.831458833 podStartE2EDuration="2.831458833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.826898108 +0000 UTC m=+1302.810804988" watchObservedRunningTime="2026-10-30 22:04:53.831458833 +0000 UTC m=+1302.815365740"
Oct 30 22:04:54.108860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:54.106576    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:54.108860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:54.106634    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:54.139437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:54.109657    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" podStartSLOduration=3.10963885 podStartE2EDuration="3.10963885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:54.104840091 +0000 UTC m=+1303.088746964" watchObservedRunningTime="2026-10-30 22:04:54.10963885 +0000 UTC m=+1303.093545727"
Oct 30 22:04:55.090849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.072186    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:55.109888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.109853    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:55.127766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.112048    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:55.127766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.112254    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:55.127766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.116729    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:55.212523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.212488    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\" (UniqueName: \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\") pod \"8bf16d92-b028-48ba-883c-c545d3dc64bd\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") "
Oct 30 22:04:55.212965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.212940    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\") pod \"8bf16d92-b028-48ba-883c-c545d3dc64bd\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") "
Oct 30 22:04:55.214613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.214590    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\") pod \"8bf16d92-b028-48ba-883c-c545d3dc64bd\" (UID: \"8bf16d92-b028-48ba-883c-c545d3dc64bd\") "
Oct 30 22:04:55.214912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.213436    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca" pod "8bf16d92-b028-48ba-883c-c545d3dc64bd" (UID: "8bf16d92-b028-48ba-883c-c545d3dc64bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:55.216125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.216101    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm" pod "8bf16d92-b028-48ba-883c-c545d3dc64bd" (UID: "8bf16d92-b028-48ba-883c-c545d3dc64bd"). InnerVolumeSpecName "kube-api-access-cpdwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:55.217522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.217493    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdwm\" (UniqueName: \"kubernetes.io/projected/8bf16d92-b028-48ba-883c-c545d3dc64bd-kube-api-access-cpdwm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.217666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.217651    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.218201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.218180    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca" pod "8bf16d92-b028-48ba-883c-c545d3dc64bd" (UID: "8bf16d92-b028-48ba-883c-c545d3dc64bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:55.322053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.321976    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\") pod \"12959834-6b5c-4fef-9d48-c58f4dec0314\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") "
Oct 30 22:04:55.322053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.322021    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\") pod \"12959834-6b5c-4fef-9d48-c58f4dec0314\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") "
Oct 30 22:04:55.322368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.322346    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\" (UniqueName: \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\") pod \"12959834-6b5c-4fef-9d48-c58f4dec0314\" (UID: \"12959834-6b5c-4fef-9d48-c58f4dec0314\") "
Oct 30 22:04:55.322472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.322375    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca" pod "12959834-6b5c-4fef-9d48-c58f4dec0314" (UID: "12959834-6b5c-4fef-9d48-c58f4dec0314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:55.323144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.323125    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.323144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.323145    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf16d92-b028-48ba-883c-c545d3dc64bd-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.324164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.324141    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca" pod "12959834-6b5c-4fef-9d48-c58f4dec0314" (UID: "12959834-6b5c-4fef-9d48-c58f4dec0314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:55.325413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.325381    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j" pod "12959834-6b5c-4fef-9d48-c58f4dec0314" (UID: "12959834-6b5c-4fef-9d48-c58f4dec0314"). InnerVolumeSpecName "kube-api-access-kg62j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:55.424642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.424607    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12959834-6b5c-4fef-9d48-c58f4dec0314-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.424642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.424638    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg62j\" (UniqueName: \"kubernetes.io/projected/12959834-6b5c-4fef-9d48-c58f4dec0314-kube-api-access-kg62j\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:55.737018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.736988    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"
Oct 30 22:04:55.737192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.737039    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp" event={"ID":"12959834-6b5c-4fef-9d48-c58f4dec0314","Type":"ContainerDied","Data":"6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65"}
Oct 30 22:04:55.737192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.737101    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b94ae5ced48e42587c27b09cd1b1e38b209ba776502215b972ea2d504761b65"
Oct 30 22:04:55.738567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.738547    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"
Oct 30 22:04:55.738703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.738578    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r" event={"ID":"8bf16d92-b028-48ba-883c-c545d3dc64bd","Type":"ContainerDied","Data":"c05af6b42f711f4655467a32b86b5312593dea4e0f9b0e5689afcad0c13bbacd"}
Oct 30 22:04:55.738703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:55.738614    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05af6b42f711f4655467a32b86b5312593dea4e0f9b0e5689afcad0c13bbacd"
Oct 30 22:04:56.092685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:56.092595    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:04:56.093170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:56.093138    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 22:04:56.337902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:56.337861    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:57.097606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:57.097573    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:57.099052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:57.099022    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:04:57.615558 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:04:57.615522    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:04:58.833811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.833768    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"]
Oct 30 22:04:58.834549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834526    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8bf16d92-b028-48ba-883c-c545d3dc64bd" containerName="revoke-keys"
Oct 30 22:04:58.834549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834547    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf16d92-b028-48ba-883c-c545d3dc64bd" containerName="revoke-keys"
Oct 30 22:04:58.834709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834575    5510 cpu_manager.go:414] "containerMap: removing container" podUID="12959834-6b5c-4fef-9d48-c58f4dec0314" containerName="revoke-keys"
Oct 30 22:04:58.834709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834582    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12959834-6b5c-4fef-9d48-c58f4dec0314" containerName="revoke-keys"
Oct 30 22:04:58.834804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834719    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="12959834-6b5c-4fef-9d48-c58f4dec0314" containerName="revoke-keys"
Oct 30 22:04:58.834804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.834750    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf16d92-b028-48ba-883c-c545d3dc64bd" containerName="revoke-keys"
Oct 30 22:04:58.838261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.838240    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:58.890445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.890412    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"]
Oct 30 22:04:58.899890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.899862    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmmb\" (UniqueName: \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:58.900033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.900007    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:58.900224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.900196    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:58.931431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.931392    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:58.931626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.931456    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:04:58.931626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.931501    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:58.932330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:58.932314    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:59.002058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.002025    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmmb\" (UniqueName: \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.002202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.002155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.002292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.002280    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.002905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.002880    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.004434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.004411    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.055167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.055137    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmmb\" (UniqueName: \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\") pod \"maas-api-revoke-sub-3335edf6-mvpql\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.164005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.163929    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:04:59.543319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.543265    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"]
Oct 30 22:04:59.548665 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:04:59.548633    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913748d3_eb12_4a95_93fa_249cfe00183c.slice/crio-d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3 WatchSource:0}: Error finding container d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3: Status 404 returned error can't find the container with id d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3
Oct 30 22:04:59.773380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:04:59.773350    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql" event={"ID":"913748d3-eb12-4a95-93fa-249cfe00183c","Type":"ContainerStarted","Data":"d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3"}
Oct 30 22:05:00.736442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.736411    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:00.785805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.785666    5510 generic.go:356] "Generic (PLEG): container finished" podID="913748d3-eb12-4a95-93fa-249cfe00183c" containerID="139fa99be4104efba77a84a9b0c2a29f98882353e5e49d175f205bb5233ad8f1" exitCode=0
Oct 30 22:05:00.785805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.785711    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql" event={"ID":"913748d3-eb12-4a95-93fa-249cfe00183c","Type":"ContainerDied","Data":"139fa99be4104efba77a84a9b0c2a29f98882353e5e49d175f205bb5233ad8f1"}
Oct 30 22:05:00.884458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.884422    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:05:00.888884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.888861    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.936802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.932559    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.936802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.932768    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.936802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.932900    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954120    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954234    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzls\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954291    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954387    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954420    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.957806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.954458    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:00.969804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:00.966849    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3bf5aa7a-openshift-default-dockercfg-8msq4\""
Oct 30 22:05:01.032530 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:05:01.008155    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:01.032530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.010324    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057376    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057492    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057573    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057646    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057717    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzls\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057865    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057910    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.057944    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.096938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.058515    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.100288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.100097    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.100805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.100752    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.101574 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.101536    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.102207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.102174    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.142851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.139152    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.142851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.141804    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.313270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.313181    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzls\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.439988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.439933    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:01.482600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.482482    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:05:01.504760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.504371    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\") pod \"e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") " pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.534396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.534351    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:01.938100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.938074    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:05:01.977158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.977111    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\" (UniqueName: \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\") pod \"913748d3-eb12-4a95-93fa-249cfe00183c\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") "
Oct 30 22:05:01.977457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.977429    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\") pod \"913748d3-eb12-4a95-93fa-249cfe00183c\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") "
Oct 30 22:05:01.977577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.977479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\") pod \"913748d3-eb12-4a95-93fa-249cfe00183c\" (UID: \"913748d3-eb12-4a95-93fa-249cfe00183c\") "
Oct 30 22:05:01.977916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.977889    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca" pod "913748d3-eb12-4a95-93fa-249cfe00183c" (UID: "913748d3-eb12-4a95-93fa-249cfe00183c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:01.978229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.978204    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:01.979551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.979522    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca" pod "913748d3-eb12-4a95-93fa-249cfe00183c" (UID: "913748d3-eb12-4a95-93fa-249cfe00183c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:01.979643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:01.979568    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb" pod "913748d3-eb12-4a95-93fa-249cfe00183c" (UID: "913748d3-eb12-4a95-93fa-249cfe00183c"). InnerVolumeSpecName "kube-api-access-ttmmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:02.047145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.047104    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:05:02.051949 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:02.051917    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd66532_5265_43f5_a3d6_8a5db541364b.slice/crio-57a4681fe4620ee9afc8447b27a738993dd4e42f1be45eea8964debf814cd594 WatchSource:0}: Error finding container 57a4681fe4620ee9afc8447b27a738993dd4e42f1be45eea8964debf814cd594: Status 404 returned error can't find the container with id 57a4681fe4620ee9afc8447b27a738993dd4e42f1be45eea8964debf814cd594
Oct 30 22:05:02.054773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.054739    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:02.054899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.054825    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:02.054899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.054856    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:02.079798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.079757    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttmmb\" (UniqueName: \"kubernetes.io/projected/913748d3-eb12-4a95-93fa-249cfe00183c-kube-api-access-ttmmb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:02.079896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.079803    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913748d3-eb12-4a95-93fa-249cfe00183c-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:02.803365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.803331    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"
Oct 30 22:05:02.803562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.803330    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql" event={"ID":"913748d3-eb12-4a95-93fa-249cfe00183c","Type":"ContainerDied","Data":"d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3"}
Oct 30 22:05:02.803562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.803511    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a6ddb52af28e9a76f684cc3c488df648e2172e221ec5b3f2327f8b66f9fee3"
Oct 30 22:05:02.805425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.805393    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerStarted","Data":"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"}
Oct 30 22:05:02.805568 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.805450    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerStarted","Data":"57a4681fe4620ee9afc8447b27a738993dd4e42f1be45eea8964debf814cd594"}
Oct 30 22:05:02.963642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:02.963574    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podStartSLOduration=2.963552436 podStartE2EDuration="2.963552436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:02.961650775 +0000 UTC m=+1311.945557648" watchObservedRunningTime="2026-10-30 22:05:02.963552436 +0000 UTC m=+1311.947459313"
Oct 30 22:05:03.347546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.347497    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:03.535104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.535069    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:03.540439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.540416    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:03.843518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.843486    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:03.844702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.844673    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:05:03.906448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.906418    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:05:03.907423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.907406    5510 cpu_manager.go:414] "containerMap: removing container" podUID="913748d3-eb12-4a95-93fa-249cfe00183c" containerName="revoke-keys"
Oct 30 22:05:03.907501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.907425    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913748d3-eb12-4a95-93fa-249cfe00183c" containerName="revoke-keys"
Oct 30 22:05:03.907639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.907627    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="913748d3-eb12-4a95-93fa-249cfe00183c" containerName="revoke-keys"
Oct 30 22:05:03.912149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.912115    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:03.967394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:03.961998    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-50ff7514-openshift-default-dockercfg-g859w\""
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.052516    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067286    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067333    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067360    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067421    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067550    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067633    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfg75\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067773    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067856    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.083928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.067928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170155    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170203    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170231    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170284    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170393    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170464    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfg75\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170579    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170632    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.170848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.170692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.174040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.173940    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.174361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.174220    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.178804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.177894    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.178804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.178265    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.203057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.183288    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.203057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.183829    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.203523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.203448    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.361992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.361924    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfg75\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.393181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.391132    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:04.399932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.399864    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.405134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.404725    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:05:04.696571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.696528    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:04.730202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.730171    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:05:04.731508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:04.731485    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:05:05.099025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.098935    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:05.172562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.172524    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:05:05.177716 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:05.177655    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b515f9_9ff1_4884_b543_4c138abc4258.slice/crio-ad6999e056255fd9528d2e7261af7b7f4fb668bd9d27567543fbd8bd5c850b8a WatchSource:0}: Error finding container ad6999e056255fd9528d2e7261af7b7f4fb668bd9d27567543fbd8bd5c850b8a: Status 404 returned error can't find the container with id ad6999e056255fd9528d2e7261af7b7f4fb668bd9d27567543fbd8bd5c850b8a
Oct 30 22:05:05.181870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.181830    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:05.181971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.181926    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:05.182021 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.181975    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:05:05.278919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.278897    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:05.459473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.459441    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:05.716256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.716162    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:05.863068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.863026    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerStarted","Data":"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"}
Oct 30 22:05:05.863239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.863092    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerStarted","Data":"ad6999e056255fd9528d2e7261af7b7f4fb668bd9d27567543fbd8bd5c850b8a"}
Oct 30 22:05:05.981495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:05.981393    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podStartSLOduration=2.981380498 podStartE2EDuration="2.981380498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:05.979507363 +0000 UTC m=+1314.963414239" watchObservedRunningTime="2026-10-30 22:05:05.981380498 +0000 UTC m=+1314.965287374"
Oct 30 22:05:06.441193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.441103    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:06.592230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.592195    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:06.697454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.697367    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:06.702690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.702665    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:06.873382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.873352    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:06.874424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:06.874403    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:05:07.137274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:07.137232    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:07.347173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:07.347142    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:07.705549 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:05:07.705514    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e49b28_2047_4351_9f85_60a31a5d91ba.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:07.840736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:07.840698    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:08.232091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.232051    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:08.415511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.415476    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:08.849171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.842908    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:05:08.867804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.862406    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:05:08.867804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.862576    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:08.917800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.916960    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3bf5aa7a\""
Oct 30 22:05:08.990842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.962431    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:08.990842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:08.962474    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5hm\" (UniqueName: \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.050804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.049081    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3bf5aa7a-dockercfg-mnvh5\""
Oct 30 22:05:09.157854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.096177    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:05:09.157854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.116977    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.157854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.117027    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5hm\" (UniqueName: \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.157854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.129618    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.157854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.133672    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:09.262828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.236117    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.262828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.258491    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.262828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.258716    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwlz\" (UniqueName: \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.329978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.329939    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5hm\" (UniqueName: \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\") pod \"payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") " pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.336372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.336343    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:05:09.351347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.351311    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:09.361826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.360684    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.361826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.361601    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwlz\" (UniqueName: \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.362616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.362593    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.415993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.415919    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwlz\" (UniqueName: \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\") pod \"payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.519822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.519789    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:09.631474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.631435    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:09.898750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:09.898717    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:05:09.903983 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:09.903956    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bec354_5da0_4084_a894_982f611629fe.slice/crio-bc68d0a8f0f16c44847f0aac42f325c4cfa5cdb1077c37e7bf8b6e50f3979443 WatchSource:0}: Error finding container bc68d0a8f0f16c44847f0aac42f325c4cfa5cdb1077c37e7bf8b6e50f3979443: Status 404 returned error can't find the container with id bc68d0a8f0f16c44847f0aac42f325c4cfa5cdb1077c37e7bf8b6e50f3979443
Oct 30 22:05:10.027710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.027682    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:05:10.030249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.030218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" event={"ID":"90bec354-5da0-4084-a894-982f611629fe","Type":"ContainerStarted","Data":"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"}
Oct 30 22:05:10.030373 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:10.030264    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5826f62_7612_4d90_88a1_daf37f55742f.slice/crio-d7ece68c1bcd8d20bcba7241b450a36006a0c531537c25ece79725b0aba176a1 WatchSource:0}: Error finding container d7ece68c1bcd8d20bcba7241b450a36006a0c531537c25ece79725b0aba176a1: Status 404 returned error can't find the container with id d7ece68c1bcd8d20bcba7241b450a36006a0c531537c25ece79725b0aba176a1
Oct 30 22:05:10.030373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.030273    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" event={"ID":"90bec354-5da0-4084-a894-982f611629fe","Type":"ContainerStarted","Data":"bc68d0a8f0f16c44847f0aac42f325c4cfa5cdb1077c37e7bf8b6e50f3979443"}
Oct 30 22:05:10.030543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.030525    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:10.146937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.146887    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" podStartSLOduration=2.146868604 podStartE2EDuration="2.146868604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:10.141471348 +0000 UTC m=+1319.125378223" watchObservedRunningTime="2026-10-30 22:05:10.146868604 +0000 UTC m=+1319.130775481"
Oct 30 22:05:10.712186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:10.712154    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:11.042665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:11.042574    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" event={"ID":"c5826f62-7612-4d90-88a1-daf37f55742f","Type":"ContainerStarted","Data":"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"}
Oct 30 22:05:11.043107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:11.042671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" event={"ID":"c5826f62-7612-4d90-88a1-daf37f55742f","Type":"ContainerStarted","Data":"d7ece68c1bcd8d20bcba7241b450a36006a0c531537c25ece79725b0aba176a1"}
Oct 30 22:05:11.043107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:11.042802    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:11.163799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:11.163727    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" podStartSLOduration=3.163709121 podStartE2EDuration="3.163709121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:11.158825717 +0000 UTC m=+1320.142732592" watchObservedRunningTime="2026-10-30 22:05:11.163709121 +0000 UTC m=+1320.147616000"
Oct 30 22:05:12.077845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:12.077811    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:12.240067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:12.240031    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:13.281844 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:05:13.281768    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:05:13.431420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:13.431388    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:13.610123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:13.610017    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:14.794214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:14.794179    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:14.958977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:14.958938    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:15.418234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:15.418203    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:15.594975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:15.594939    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:15.816809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:15.803446    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:15.944601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:15.941043    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:05:16.003625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.003533    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:16.140920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.131172    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:05:16.140920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.132604    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.219852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.219798    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50ff7514\""
Oct 30 22:05:16.227838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.225056    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50ff7514-dockercfg-gzc54\""
Oct 30 22:05:16.253635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.253600    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:05:16.284741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.259934    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:05:16.293407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.281665    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.329849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.329810    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.330274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.330249    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.348620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.348579    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42psp\" (UniqueName: \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.360836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.349478    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6x79\" (UniqueName: \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.578804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.567227    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.579223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.579186    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42psp\" (UniqueName: \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.579533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.579499    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6x79\" (UniqueName: \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.579947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.579926    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.592200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.589707    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.614840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.602916    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.651825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.647434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42psp\" (UniqueName: \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\") pod \"payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.651825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.648873    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6x79\" (UniqueName: \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\") pod \"payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") " pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:16.702535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.702502    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:16.722793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:16.722758    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:17.096868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.096791    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:05:17.100664 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:17.100632    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33287dd_ab87_4985_ac1e_9aa1a8c02c39.slice/crio-bd5674e7f35d23ed518067db8b6eee6d9f1e3a151d1a243926b99dcabb8f66e7 WatchSource:0}: Error finding container bd5674e7f35d23ed518067db8b6eee6d9f1e3a151d1a243926b99dcabb8f66e7: Status 404 returned error can't find the container with id bd5674e7f35d23ed518067db8b6eee6d9f1e3a151d1a243926b99dcabb8f66e7
Oct 30 22:05:17.142883 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.142796    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:05:17.157251 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:17.157227    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7ce8ef_8c22_4f13_8a13_8eb10956720e.slice/crio-4e53e0cadcaecca8f895704b0171adf7aef87905e1654a198feff6e41264f5cd WatchSource:0}: Error finding container 4e53e0cadcaecca8f895704b0171adf7aef87905e1654a198feff6e41264f5cd: Status 404 returned error can't find the container with id 4e53e0cadcaecca8f895704b0171adf7aef87905e1654a198feff6e41264f5cd
Oct 30 22:05:17.349010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.348919    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" event={"ID":"ca7ce8ef-8c22-4f13-8a13-8eb10956720e","Type":"ContainerStarted","Data":"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"}
Oct 30 22:05:17.349010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.348978    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" event={"ID":"ca7ce8ef-8c22-4f13-8a13-8eb10956720e","Type":"ContainerStarted","Data":"4e53e0cadcaecca8f895704b0171adf7aef87905e1654a198feff6e41264f5cd"}
Oct 30 22:05:17.349010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.349009    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:17.350732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.350705    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" event={"ID":"c33287dd-ab87-4985-ac1e-9aa1a8c02c39","Type":"ContainerStarted","Data":"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"}
Oct 30 22:05:17.350867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.350751    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" event={"ID":"c33287dd-ab87-4985-ac1e-9aa1a8c02c39","Type":"ContainerStarted","Data":"bd5674e7f35d23ed518067db8b6eee6d9f1e3a151d1a243926b99dcabb8f66e7"}
Oct 30 22:05:17.350867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.350794    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:17.455887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.455833    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" podStartSLOduration=2.455819426 podStartE2EDuration="2.455819426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:17.454271633 +0000 UTC m=+1326.438178511" watchObservedRunningTime="2026-10-30 22:05:17.455819426 +0000 UTC m=+1326.439726299"
Oct 30 22:05:17.563008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:17.562947    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" podStartSLOduration=2.562931958 podStartE2EDuration="2.562931958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:17.561468876 +0000 UTC m=+1326.545375752" watchObservedRunningTime="2026-10-30 22:05:17.562931958 +0000 UTC m=+1326.546838833"
Oct 30 22:05:19.865530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:19.865488    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:20.043519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:20.043489    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:20.314028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:20.313996    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:20.767128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:20.767097    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:20.961896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:20.961859    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"]
Oct 30 22:05:20.967368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:20.967344    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.019186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.019116    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"]
Oct 30 22:05:21.045675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.045647    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:05:21.059460 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.059430    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:21.059624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.059489    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:21.061614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.061596    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:21.061687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.061656    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:05:21.103309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.103273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbmq\" (UniqueName: \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.103475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.103396    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.103631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.103614    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.210121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.210001    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.210329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.210296    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.210542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.210521    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbmq\" (UniqueName: \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.212244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.212182    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.213993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.213959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.264865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.264841    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbmq\" (UniqueName: \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\") pod \"maas-api-revoke-sub-ea30ba4d-xnwdc\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.285944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.285879    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:21.687439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.687357    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"]
Oct 30 22:05:21.690982 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:05:21.690956    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58878a49_5c1c_4736_aac3_9a27a4b0694e.slice/crio-4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95 WatchSource:0}: Error finding container 4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95: Status 404 returned error can't find the container with id 4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95
Oct 30 22:05:21.732386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.732359    5510 scope.go:122] "RemoveContainer" containerID="e9a0a1ef6ea925f98205b7860d3e9d234b390118d044dc4ebf5612eee78e9de5"
Oct 30 22:05:21.743001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.742976    5510 scope.go:122] "RemoveContainer" containerID="a6fb707cbe067994cfb7bf7c3a8aa82756bc0a1dd163e295c04b1e016b9f0d50"
Oct 30 22:05:21.752921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.752896    5510 scope.go:122] "RemoveContainer" containerID="030e8105ebbebe0e2ce5570d1a1a7b8a1fbd51fbcdc14d279072bd9c91764d34"
Oct 30 22:05:21.764216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.764191    5510 scope.go:122] "RemoveContainer" containerID="988ece0732c241738e1cd347566a58155b654e5d33a9e978ee974fe46597e690"
Oct 30 22:05:21.778152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.778132    5510 scope.go:122] "RemoveContainer" containerID="feb13e84a1762b70f0616f35bb7b64ce72d36e45b3062e5a6319281bf9c85eeb"
Oct 30 22:05:21.791060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.791040    5510 scope.go:122] "RemoveContainer" containerID="9e455e7a5495d02ee3eedbd887ba8d13aa85fdacebd77eddead6f0b4d08242ec"
Oct 30 22:05:21.801438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.801412    5510 scope.go:122] "RemoveContainer" containerID="4dd5492822173d9bd1f11334437277d63ae1e80367e0a9d89365054768721da5"
Oct 30 22:05:21.810768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.810745    5510 scope.go:122] "RemoveContainer" containerID="83e05121e5fe7f7efe7627d367a54bee5c8d90fd0028bc4ba472fad946c04816"
Oct 30 22:05:21.819907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:21.819889    5510 scope.go:122] "RemoveContainer" containerID="464a438bdb8a87fe382a9cc3e37de8bb777f437c950937d6e48c79fe689216d6"
Oct 30 22:05:22.054907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:22.054877    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:05:22.410413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:22.410320    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc" event={"ID":"58878a49-5c1c-4736-aac3-9a27a4b0694e","Type":"ContainerStarted","Data":"28ead4fdcd0c737479272e92d0f1a287b72815d7631cf2aeca80e6ba666ed55f"}
Oct 30 22:05:22.410413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:22.410406    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc" event={"ID":"58878a49-5c1c-4736-aac3-9a27a4b0694e","Type":"ContainerStarted","Data":"4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95"}
Oct 30 22:05:22.526185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:22.526135    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc" podStartSLOduration=2.526122283 podStartE2EDuration="2.526122283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:22.523540449 +0000 UTC m=+1331.507447338" watchObservedRunningTime="2026-10-30 22:05:22.526122283 +0000 UTC m=+1331.510029158"
Oct 30 22:05:23.455804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:23.451128    5510 generic.go:356] "Generic (PLEG): container finished" podID="58878a49-5c1c-4736-aac3-9a27a4b0694e" containerID="28ead4fdcd0c737479272e92d0f1a287b72815d7631cf2aeca80e6ba666ed55f" exitCode=0
Oct 30 22:05:23.455804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:23.451421    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc" event={"ID":"58878a49-5c1c-4736-aac3-9a27a4b0694e","Type":"ContainerDied","Data":"28ead4fdcd0c737479272e92d0f1a287b72815d7631cf2aeca80e6ba666ed55f"}
Oct 30 22:05:24.492859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.415436    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:24.772003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.771903    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:24.850958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.850904    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:24.866238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.866190    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:05:24.977130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.977090    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:24.985946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.985899    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:24.993718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:24.986176    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.044895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.044802    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.109934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.109890    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.172808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.161754    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.243637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.243589    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.325039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.324903    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:05:25.325661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.325623    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.795805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.781915    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.795805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.782511    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:05:25.876807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.875814    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:25.876807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.876241    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.979874    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.988867    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.989487    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:25.989548    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.002872    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.003404    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.056588    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.057166    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.153746    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.154058    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.169852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.155574    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:05:26.246803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.182365    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.246803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.183147    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:05:26.277820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.255171    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.277820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.255662    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:05:26.312806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.308682    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:26.341520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.341470    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:26.402310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.368747    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:05:26.713162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.694041    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:26.738597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.738561    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc" event={"ID":"58878a49-5c1c-4736-aac3-9a27a4b0694e","Type":"ContainerDied","Data":"4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95"}
Oct 30 22:05:26.749091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.749062    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f665cecd8f35642627a815fcffeb4ca7feb9e1dd8c86b1be2bf572f9357af95"
Oct 30 22:05:26.840801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.840734    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:05:26.885932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:26.860925    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:05:27.085807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.081843    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.191098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.122483    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.219539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.181879    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.251799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.251734    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:27.252379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.252354    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.309500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.309460    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:27.412067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.412019    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.630084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.630029    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:27.654912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.654866    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.679718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.679676    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:05:27.926871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:27.901998    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.156810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.156696    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:28.230168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.209858    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.267542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.267498    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.316646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.269281    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:05:28.323866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.313734    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:05:28.324396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.314509    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\" (UniqueName: \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\") pod \"58878a49-5c1c-4736-aac3-9a27a4b0694e\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") "
Oct 30 22:05:28.324647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.324629    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\") pod \"58878a49-5c1c-4736-aac3-9a27a4b0694e\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") "
Oct 30 22:05:28.363710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.363665    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca" pod "58878a49-5c1c-4736-aac3-9a27a4b0694e" (UID: "58878a49-5c1c-4736-aac3-9a27a4b0694e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:28.435921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.435863    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.437558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.437523    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\") pod \"58878a49-5c1c-4736-aac3-9a27a4b0694e\" (UID: \"58878a49-5c1c-4736-aac3-9a27a4b0694e\") "
Oct 30 22:05:28.469804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.438393    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:28.483511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.479927    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:05:28.492261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.492215    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca" pod "58878a49-5c1c-4736-aac3-9a27a4b0694e" (UID: "58878a49-5c1c-4736-aac3-9a27a4b0694e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:28.524399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.524365    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:05:28.592804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.584881    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58878a49-5c1c-4736-aac3-9a27a4b0694e-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:28.615963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.615919    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.683345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.683302    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.684479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.684446    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:05:28.689056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.689029    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.732534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.732481    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:28.791671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.760734    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:05:28.903088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:28.903043    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.187836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.177844    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded"
Oct 30 22:05:29.217797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.217735    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq" pod "58878a49-5c1c-4736-aac3-9a27a4b0694e" (UID: "58878a49-5c1c-4736-aac3-9a27a4b0694e"). InnerVolumeSpecName "kube-api-access-hwbmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.251595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.251542    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.416134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.416030    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.440396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.440354    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.451144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.445381    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:05:29.508635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.445436    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:05:29.638148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.445476    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.638148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.528462    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbmq\" (UniqueName: \"kubernetes.io/projected/58878a49-5c1c-4736-aac3-9a27a4b0694e-kube-api-access-hwbmq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:29.638148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.550086    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.638148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.550174    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.658589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.658543    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"
Oct 30 22:05:29.684856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.680948    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:05:29.793123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.793076    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:05:29.847808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:29.831863    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:05:30.116281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.036190    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:05:30.116939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.116876    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:05:30.117624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.092768    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:05:30.263123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.263069    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:30.496813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.491334    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:30.515802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.491105    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:05:30.587804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.587410    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:05:30.603870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.603833    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:05:30.630532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.630503    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:05:30.686865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.685826    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:05:30.733130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.733055    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:05:30.948028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.908042    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:30.948028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:30.917989    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:05:31.124660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.124621    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:31.130122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.119961    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:05:31.152518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.151964    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:05:31.152518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.152021    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:05:31.152518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.152052    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:05:31.152518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.152082    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:05:31.153716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.153350    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:05:31.572299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.572267    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:31.743131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.743095    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:31.943164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:31.943082    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:32.974027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:32.973987    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.325961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:36.325924    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.799721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:36.799689    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:36.953494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:36.953461    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:37.892919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:37.892889    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:39.208013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:39.207979    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:40.934845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:40.934805    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:43.281836 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:05:43.281759    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:05:45.399399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:45.399368    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:49.784254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:49.784219    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:49.956802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:49.953736    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:50.114854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:50.114062    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:53.477034 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:53.476934    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" podUID="0e90b7cbbf3b5bb50f8f3af03ed08b90" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:54.370570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:54.370529    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:58.876364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:58.876322    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.033791 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:59.033738    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.201668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:59.201558    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.656134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:59.656106    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:59.801958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:05:59.801913    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:01.289241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:01.289200    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:06:03.552273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:03.552235    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:09.267135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:09.267084    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:12.605935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:12.605901    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:13.282441 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:06:13.282404    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:06:16.994081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:16.994046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:21.465193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:21.465149    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:24.145154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.145113    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"]
Oct 30 22:06:24.147001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.146973    5510 cpu_manager.go:414] "containerMap: removing container" podUID="58878a49-5c1c-4736-aac3-9a27a4b0694e" containerName="revoke-keys"
Oct 30 22:06:24.159767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.159720    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58878a49-5c1c-4736-aac3-9a27a4b0694e" containerName="revoke-keys"
Oct 30 22:06:24.160433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.160404    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="58878a49-5c1c-4736-aac3-9a27a4b0694e" containerName="revoke-keys"
Oct 30 22:06:24.167104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.167072    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.199466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.199433    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"]
Oct 30 22:06:24.215686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.215650    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.216019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.215992    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrhb\" (UniqueName: \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.216202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.216183    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.238074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.238044    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:24.238226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.238070    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:24.238345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.238127    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:24.238345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.238231    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:06:24.317983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.317950    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.318177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.318026    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrhb\" (UniqueName: \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.318177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.318065    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.318775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.318751    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.320508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.320475    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.371474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.371440    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrhb\" (UniqueName: \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\") pod \"maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.484023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.483990    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:24.534619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.534583    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:06:24.539886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.539860    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:06:24.540046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.539993    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.551710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.551057    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623059    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623100    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623123    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623190    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623244    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623266    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623280    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623297    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623529    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllfs\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623572    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623610    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623654    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623690    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.623867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623759    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623806    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.623867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623836    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntb5r\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.624115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623871    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.624115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.623903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.629006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.628977    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:06:24.675244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.675203    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 22:06:24.696016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.695986    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:06:24.696189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.696019    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:24.725829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.725794    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.725835    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.725867    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntb5r\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.725904    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.725937    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726059    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726093    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726127    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726403    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726452    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726529    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726536    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726610    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726638    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726669    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726744    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726815    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllfs\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.726954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726842    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726893    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726924    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.726954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726942    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.727180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.726981    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.727180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.727017    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.728200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.727558    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.728200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.727507    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.728200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.727848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.728200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.728117    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.728486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.728362    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.730591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.730003    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.730591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.730072    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.730591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.730368    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.731830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.731807    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.780622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.780533    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.780819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.780637    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllfs\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\") pod \"maas-default-gateway-openshift-default-694db55c87-x8gnm\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.780819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.780719    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntb5r\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.780945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.780926    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-26m5d\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:24.870343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.870306    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"]
Oct 30 22:06:24.873223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.873198    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:24.875128 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:24.875102    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dc6b43_deb9_428e_bf8b_065b276d690c.slice/crio-9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af WatchSource:0}: Error finding container 9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af: Status 404 returned error can't find the container with id 9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af
Oct 30 22:06:24.879702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:24.879678    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:25.316652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.316604    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:06:25.323181 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:25.322524    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbb86f1_2ca4_49d1_b20a_6213aad11600.slice/crio-fbd91e485c6e3edb848b57c4efc59d7cdca3979bd89cfc75644475f081d59f33 WatchSource:0}: Error finding container fbd91e485c6e3edb848b57c4efc59d7cdca3979bd89cfc75644475f081d59f33: Status 404 returned error can't find the container with id fbd91e485c6e3edb848b57c4efc59d7cdca3979bd89cfc75644475f081d59f33
Oct 30 22:06:25.323422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.323400    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:06:25.324318 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:25.324291    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c7bb10_d5a0_4326_971c_ade1c0c1f046.slice/crio-3be860c30673cc344dea3a6af51a7792e311e6cccd4b1e8f9e6fbeef653025ee WatchSource:0}: Error finding container 3be860c30673cc344dea3a6af51a7792e311e6cccd4b1e8f9e6fbeef653025ee: Status 404 returned error can't find the container with id 3be860c30673cc344dea3a6af51a7792e311e6cccd4b1e8f9e6fbeef653025ee
Oct 30 22:06:25.325055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.325011    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.325153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.325093    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.325153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.325138    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.327265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.327234    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.327872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.327304    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.327872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.327349    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:25.410283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.410251    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf" event={"ID":"31dc6b43-deb9-428e-bf8b-065b276d690c","Type":"ContainerStarted","Data":"44dfcf31534480a3c6566b51bd499a546186516174ba7fba0c6558638fe91748"}
Oct 30 22:06:25.410448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.410312    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf" event={"ID":"31dc6b43-deb9-428e-bf8b-065b276d690c","Type":"ContainerStarted","Data":"9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af"}
Oct 30 22:06:25.411926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.411895    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" event={"ID":"4bbb86f1-2ca4-49d1-b20a-6213aad11600","Type":"ContainerStarted","Data":"fbd91e485c6e3edb848b57c4efc59d7cdca3979bd89cfc75644475f081d59f33"}
Oct 30 22:06:25.413603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.413577    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" event={"ID":"21c7bb10-d5a0-4326-971c-ade1c0c1f046","Type":"ContainerStarted","Data":"3be860c30673cc344dea3a6af51a7792e311e6cccd4b1e8f9e6fbeef653025ee"}
Oct 30 22:06:25.517100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:25.517037    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf" podStartSLOduration=1.517019101 podStartE2EDuration="1.517019101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:25.516310614 +0000 UTC m=+1394.500217490" watchObservedRunningTime="2026-10-30 22:06:25.517019101 +0000 UTC m=+1394.500925977"
Oct 30 22:06:26.306582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.306241    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:26.520382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.520346    5510 generic.go:356] "Generic (PLEG): container finished" podID="31dc6b43-deb9-428e-bf8b-065b276d690c" containerID="44dfcf31534480a3c6566b51bd499a546186516174ba7fba0c6558638fe91748" exitCode=0
Oct 30 22:06:26.521170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.521125    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf" event={"ID":"31dc6b43-deb9-428e-bf8b-065b276d690c","Type":"ContainerDied","Data":"44dfcf31534480a3c6566b51bd499a546186516174ba7fba0c6558638fe91748"}
Oct 30 22:06:26.554908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.554856    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" event={"ID":"4bbb86f1-2ca4-49d1-b20a-6213aad11600","Type":"ContainerStarted","Data":"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"}
Oct 30 22:06:26.618865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.598459    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" event={"ID":"21c7bb10-d5a0-4326-971c-ade1c0c1f046","Type":"ContainerStarted","Data":"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"}
Oct 30 22:06:26.876260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.875175    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:26.893203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.881909    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:26.893203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.881979    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:26.893203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.882435    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:26.909805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.893921    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:26.909805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.894003    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:26.917843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.916946    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podStartSLOduration=2.916923227 podStartE2EDuration="2.916923227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.796179292 +0000 UTC m=+1395.780086170" watchObservedRunningTime="2026-10-30 22:06:26.916923227 +0000 UTC m=+1395.900830104"
Oct 30 22:06:26.921738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:26.921691    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podStartSLOduration=2.921677067 podStartE2EDuration="2.921677067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.908775284 +0000 UTC m=+1395.892682158" watchObservedRunningTime="2026-10-30 22:06:26.921677067 +0000 UTC m=+1395.905583999"
Oct 30 22:06:27.874348 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:27.874252    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:27.878801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:27.874925    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:27.896024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:27.894276    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:27.907026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:27.906987    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:28.012842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.012235    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:28.078807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.078154    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\") pod \"31dc6b43-deb9-428e-bf8b-065b276d690c\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") "
Oct 30 22:06:28.078807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.078341    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\") pod \"31dc6b43-deb9-428e-bf8b-065b276d690c\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") "
Oct 30 22:06:28.078807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.078395    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\" (UniqueName: \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\") pod \"31dc6b43-deb9-428e-bf8b-065b276d690c\" (UID: \"31dc6b43-deb9-428e-bf8b-065b276d690c\") "
Oct 30 22:06:28.100806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.088949    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca" pod "31dc6b43-deb9-428e-bf8b-065b276d690c" (UID: "31dc6b43-deb9-428e-bf8b-065b276d690c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:28.110811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.108048    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb" pod "31dc6b43-deb9-428e-bf8b-065b276d690c" (UID: "31dc6b43-deb9-428e-bf8b-065b276d690c"). InnerVolumeSpecName "kube-api-access-pzrhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:28.110811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.109908    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca" pod "31dc6b43-deb9-428e-bf8b-065b276d690c" (UID: "31dc6b43-deb9-428e-bf8b-065b276d690c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:28.195804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.181409    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:28.195804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.181455    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dc6b43-deb9-428e-bf8b-065b276d690c-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:28.195804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.181471    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzrhb\" (UniqueName: \"kubernetes.io/projected/31dc6b43-deb9-428e-bf8b-065b276d690c-kube-api-access-pzrhb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:28.631683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.631648    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"
Oct 30 22:06:28.631683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.631671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf" event={"ID":"31dc6b43-deb9-428e-bf8b-065b276d690c","Type":"ContainerDied","Data":"9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af"}
Oct 30 22:06:28.631933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.631727    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9175961fb4b6777004a3d4366a5b295e20080a1a9d0696bc638c2effe20110af"
Oct 30 22:06:28.874595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.874557    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:28.875295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.875263    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:28.881149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.881123    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:28.881345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:28.881304    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:29.888805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:29.882123    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:29.888805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:29.882183    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:29.888805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:29.882791    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:29.888805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:29.882833    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:30.885438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:30.885405    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:30.885940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:30.885910    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:30.886444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:30.885849    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:30.886599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:30.886577    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:30.934829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:30.934756    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:31.874623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:31.874575    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:31.874863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:31.874645    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:31.881057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:31.881025    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:31.881216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:31.881095    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:32.874658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:32.874609    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:06:32.875208 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:32.874708    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:06:32.880428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:32.880392    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-26m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 22:06:32.880579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:32.880449    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 22:06:34.054461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.054405    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"]
Oct 30 22:06:34.110748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.110711    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fa0cd0-lzkts"]
Oct 30 22:06:34.273225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.273191    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:34.275059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.275035    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:34.275765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.275742    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:06:34.316246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.316171    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:34.316645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.316613    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:34.317538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:34.317492    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:06:35.128505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:35.128469    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6619e8ef-27af-4283-9b7d-5686714b55e0" path="/var/lib/kubelet/pods/6619e8ef-27af-4283-9b7d-5686714b55e0/volumes"
Oct 30 22:06:35.361512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:35.361476    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:35.605546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:35.605508    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:06:35.707537 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:35.707501    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:06:35.964958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:35.964922    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:36.118250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:36.118215    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:39.710873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.710834    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:06:39.715803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.712664    5510 cpu_manager.go:414] "containerMap: removing container" podUID="31dc6b43-deb9-428e-bf8b-065b276d690c" containerName="revoke-keys"
Oct 30 22:06:39.716015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.715991    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dc6b43-deb9-428e-bf8b-065b276d690c" containerName="revoke-keys"
Oct 30 22:06:39.733805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.716422    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dc6b43-deb9-428e-bf8b-065b276d690c" containerName="revoke-keys"
Oct 30 22:06:39.818010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.809676    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.888941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.880720    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.907666    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.907812    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.907852    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.907938    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhcv\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.907984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.908238    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.908354    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.908480    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.908637    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:39.917738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.909075    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:06:39.959502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.954768    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:39.959502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:39.959363    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:06:40.000828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.000796    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:06:40.012741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.011156    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.013583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.013558    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.014391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.014342    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.015014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.014986    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.015708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.015688    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.016363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.016344    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.017398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.017376    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.018027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.014829    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.018661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.017145    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.018854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.013363    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.018854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.015391    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.018854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.018588    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.019554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.017993    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.019792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.019756    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhcv\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.030580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.027112    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.030580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.028305    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.082806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.080771    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.083334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.083306    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhcv\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\") pod \"maas-default-gateway-openshift-default-694db55c87-blpfn\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135091    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135270    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135510    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmt9z\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135550    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135610    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135706    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135744    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.135823    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.142710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.136015    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.222803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.208941    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.241913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242013    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242157    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmt9z\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242188    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242239    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242317    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242348    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242396    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.250463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.242475    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.274811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.274485    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.305976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.305378    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.309275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.306514    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.309486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.306709    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.309611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.309362    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.309706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.309475    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.309808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.309647    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:40.311434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.311378    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.331198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.331170    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmt9z\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.369896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.369641    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-qvb8t\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.465805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.464969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:40.613276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.612897    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" podUID="9b210c05-2ead-4555-9b40-e55b0aa3be59" containerName="payload-processing" containerID="cri-o://d7f82528ed77d8b3d1e57023832274c9800e58a33769ab26219a05d4586f7b9e" gracePeriod=25
Oct 30 22:06:40.648113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.648083    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:40.708211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.708079    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" podUID="986cc2c6-8495-46d8-9e40-9227ba115da3" containerName="payload-pre-processing" containerID="cri-o://1da132428adaf7aeabfe2071aefdc5a210de32c9832c0260f6273f1a89090a71" gracePeriod=25
Oct 30 22:06:40.814679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.813682    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:06:40.815502 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:40.815458    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5290ba_2fc1_4fd3_bc6f_5d93e2f7c581.slice/crio-a8d2f7a00bdc7544bf66a9a353626982031d785256115d92c867c61879bfcf15 WatchSource:0}: Error finding container a8d2f7a00bdc7544bf66a9a353626982031d785256115d92c867c61879bfcf15: Status 404 returned error can't find the container with id a8d2f7a00bdc7544bf66a9a353626982031d785256115d92c867c61879bfcf15
Oct 30 22:06:40.820555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.818978    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:40.820555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.819061    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:40.820555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:40.819099    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:41.250185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.250147    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:06:41.354993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.350123    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:41.354993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.350213    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:41.354993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.350262    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:41.424141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.412861    5510 generic.go:356] "Generic (PLEG): container finished" podID="986cc2c6-8495-46d8-9e40-9227ba115da3" containerID="1da132428adaf7aeabfe2071aefdc5a210de32c9832c0260f6273f1a89090a71" exitCode=0
Oct 30 22:06:41.424141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.413116    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" event={"ID":"986cc2c6-8495-46d8-9e40-9227ba115da3","Type":"ContainerDied","Data":"1da132428adaf7aeabfe2071aefdc5a210de32c9832c0260f6273f1a89090a71"}
Oct 30 22:06:41.435677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.435402    5510 generic.go:356] "Generic (PLEG): container finished" podID="9b210c05-2ead-4555-9b40-e55b0aa3be59" containerID="d7f82528ed77d8b3d1e57023832274c9800e58a33769ab26219a05d4586f7b9e" exitCode=0
Oct 30 22:06:41.435677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.435602    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" event={"ID":"9b210c05-2ead-4555-9b40-e55b0aa3be59","Type":"ContainerDied","Data":"d7f82528ed77d8b3d1e57023832274c9800e58a33769ab26219a05d4586f7b9e"}
Oct 30 22:06:41.441521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.439380    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:06:41.471855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.471306    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\" (UniqueName: \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\") pod \"9b210c05-2ead-4555-9b40-e55b0aa3be59\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") "
Oct 30 22:06:41.471855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.471400    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\") pod \"9b210c05-2ead-4555-9b40-e55b0aa3be59\" (UID: \"9b210c05-2ead-4555-9b40-e55b0aa3be59\") "
Oct 30 22:06:41.515806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.498219    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume" pod "9b210c05-2ead-4555-9b40-e55b0aa3be59" (UID: "9b210c05-2ead-4555-9b40-e55b0aa3be59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:41.523405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.523367    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm" pod "9b210c05-2ead-4555-9b40-e55b0aa3be59" (UID: "9b210c05-2ead-4555-9b40-e55b0aa3be59"). InnerVolumeSpecName "kube-api-access-ndlbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:41.526846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.526811    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" event={"ID":"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581","Type":"ContainerStarted","Data":"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"}
Oct 30 22:06:41.528408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.528381    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" event={"ID":"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581","Type":"ContainerStarted","Data":"a8d2f7a00bdc7544bf66a9a353626982031d785256115d92c867c61879bfcf15"}
Oct 30 22:06:41.597031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.575488    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlbm\" (UniqueName: \"kubernetes.io/projected/9b210c05-2ead-4555-9b40-e55b0aa3be59-kube-api-access-ndlbm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:41.597031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.575549    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b210c05-2ead-4555-9b40-e55b0aa3be59-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:41.636163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.635625    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:06:41.721558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.721494    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podStartSLOduration=2.721472898 podStartE2EDuration="2.721472898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:41.683358216 +0000 UTC m=+1410.667265089" watchObservedRunningTime="2026-10-30 22:06:41.721472898 +0000 UTC m=+1410.705379775"
Oct 30 22:06:41.788310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.788269    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\") pod \"986cc2c6-8495-46d8-9e40-9227ba115da3\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") "
Oct 30 22:06:41.788594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.788569    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\" (UniqueName: \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\") pod \"986cc2c6-8495-46d8-9e40-9227ba115da3\" (UID: \"986cc2c6-8495-46d8-9e40-9227ba115da3\") "
Oct 30 22:06:41.801803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.795114    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume" pod "986cc2c6-8495-46d8-9e40-9227ba115da3" (UID: "986cc2c6-8495-46d8-9e40-9227ba115da3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:41.830330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.827210    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd" pod "986cc2c6-8495-46d8-9e40-9227ba115da3" (UID: "986cc2c6-8495-46d8-9e40-9227ba115da3"). InnerVolumeSpecName "kube-api-access-qwpdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:41.907413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.905462    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986cc2c6-8495-46d8-9e40-9227ba115da3-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:41.907413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:41.905501    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpdd\" (UniqueName: \"kubernetes.io/projected/986cc2c6-8495-46d8-9e40-9227ba115da3-kube-api-access-qwpdd\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:42.225806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.213936    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:42.230178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.230144    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:42.230421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.230393    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:42.671921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.671879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9" event={"ID":"9b210c05-2ead-4555-9b40-e55b0aa3be59","Type":"ContainerDied","Data":"90351a27adb868d2e0bcf195fd6d61852467223c0c4f3d5efbf7b40cc5b8939b"}
Oct 30 22:06:42.672358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.672336    5510 scope.go:122] "RemoveContainer" containerID="d7f82528ed77d8b3d1e57023832274c9800e58a33769ab26219a05d4586f7b9e"
Oct 30 22:06:42.673405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.672237    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"
Oct 30 22:06:42.699345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.684833    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" event={"ID":"29a1e52b-8886-416a-ac49-e6a6f117559e","Type":"ContainerStarted","Data":"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"}
Oct 30 22:06:42.699605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.699579    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" event={"ID":"29a1e52b-8886-416a-ac49-e6a6f117559e","Type":"ContainerStarted","Data":"1afb0e11c004f37964f7023b689527dca5fcc344d8b55e7f6742617fc09c3a3d"}
Oct 30 22:06:42.726257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.726226    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"
Oct 30 22:06:42.750210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.750176    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k" event={"ID":"986cc2c6-8495-46d8-9e40-9227ba115da3","Type":"ContainerDied","Data":"16267f954871b762543e648af7ed3a9dd423653206dfe76485cba950b9f39efc"}
Oct 30 22:06:42.764599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.764217    5510 scope.go:122] "RemoveContainer" containerID="1da132428adaf7aeabfe2071aefdc5a210de32c9832c0260f6273f1a89090a71"
Oct 30 22:06:42.929346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:42.923071    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podStartSLOduration=3.923052001 podStartE2EDuration="3.923052001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:42.8769188 +0000 UTC m=+1411.860825678" watchObservedRunningTime="2026-10-30 22:06:42.923052001 +0000 UTC m=+1411.906958878"
Oct 30 22:06:43.054805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.043733    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:06:43.099804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.090068    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3084d7f-7877c46b78-th7r9"]
Oct 30 22:06:43.197885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.188702    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b210c05-2ead-4555-9b40-e55b0aa3be59" path="/var/lib/kubelet/pods/9b210c05-2ead-4555-9b40-e55b0aa3be59/volumes"
Oct 30 22:06:43.214803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.212163    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:43.214803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.212237    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:43.224805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.223847    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:06:43.298935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.298895    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3084d7f-7786474447-qbm9k"]
Oct 30 22:06:43.510804 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:06:43.509886    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:06:43.651468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.649957    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:43.651982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.651943    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:43.652201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.652157    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:43.693971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.693909    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:43.963152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.862310    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:43.967901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:43.940411    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.212233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.212192    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:44.212516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.212487    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:44.227997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.227886    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.228644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.228608    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:06:44.230574 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.230537    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.275327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.275286    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.314376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.314331    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.328243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.328178    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.433915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.433879    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:44.434809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.434746    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:06:44.656912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.649171    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:44.656912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.649237    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:44.703822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.697887    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:44.703822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.698385    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:06:44.942236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:44.941888    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:45.117832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.117771    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.139096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.139068    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:06:45.141755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.141713    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.142095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.142062    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:06:45.238440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.238410    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986cc2c6-8495-46d8-9e40-9227ba115da3" path="/var/lib/kubelet/pods/986cc2c6-8495-46d8-9e40-9227ba115da3/volumes"
Oct 30 22:06:45.242550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.238711    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.243035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.240614    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.250303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.240645    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.251916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.242000    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:45.291004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.243639    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:06:45.291324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.291306    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:06:45.291510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.291489    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:06:45.293401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.289129    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:45.294284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.294250    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:06:45.327889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.290953    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:45.747835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.333368    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:45.747835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.333634    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 30 22:06:45.747835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.389739    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:45.748239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.527054    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:45.748666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.715747    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:06:45.763843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.748595    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:06:45.763843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.762998    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:06:45.763843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.763070    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:06:45.781826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.779196    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:45.781826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:45.779266    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:46.123872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.123715    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.141918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.139902    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.149819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.148826    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.412859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.276651    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.412859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.279792    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.412859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.284663    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:46.412859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.400888    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:06:46.470369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.470337    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:46.500083 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.500039    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:46.897805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.873143    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.161025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.160924    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.161874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.942755    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.162320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.942912    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:06:47.162668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.989546    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.184072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.184021    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:06:47.200812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:46.899749    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.200812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.200265    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:47.277806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.252346    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.278157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.278113    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:47.333881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.325409    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.333881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.325756    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:06:47.401947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.401903    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:06:47.468265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.468220    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:47.649259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.649220    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:47.649563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.649518    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:47.927718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:47.926117    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:06:48.117808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.117376    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.162882    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.186470    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.187454    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.187745    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.211110    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.221439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.211167    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:48.313930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.288284    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:06:48.341747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.341304    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:48.341747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.341393    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:06:48.350609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.349631    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:06:48.409810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.403687    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:06:48.500858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.477928    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:06:48.579716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.573618    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 22:06:48.902805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.851128    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"]
Oct 30 22:06:48.902805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.853430    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9b210c05-2ead-4555-9b40-e55b0aa3be59" containerName="payload-processing"
Oct 30 22:06:48.902805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.853452    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b210c05-2ead-4555-9b40-e55b0aa3be59" containerName="payload-processing"
Oct 30 22:06:48.934206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.934175    5510 cpu_manager.go:414] "containerMap: removing container" podUID="986cc2c6-8495-46d8-9e40-9227ba115da3" containerName="payload-pre-processing"
Oct 30 22:06:48.941410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.941385    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986cc2c6-8495-46d8-9e40-9227ba115da3" containerName="payload-pre-processing"
Oct 30 22:06:48.942410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.942386    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="986cc2c6-8495-46d8-9e40-9227ba115da3" containerName="payload-pre-processing"
Oct 30 22:06:48.942626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.942613    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b210c05-2ead-4555-9b40-e55b0aa3be59" containerName="payload-processing"
Oct 30 22:06:48.988806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.937841    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:48.988806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.963988    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.989228    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.989522    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"]
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.989646    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:48.989709    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.009730    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:06:49.011256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.009839    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:06:49.012232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.012206    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.034709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.033282    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:06:49.040934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.040911    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:06:49.092813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.078448    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:06:49.094165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.081105    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:49.094359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.081200    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:49.146824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.143971    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.146824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.144156    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzvf\" (UniqueName: \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.146824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.144409    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.146824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.144587    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:49.242863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.242827    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.243444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.243398    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:49.247479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.246151    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.247479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.246374    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.247479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.246476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzvf\" (UniqueName: \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.255247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.255168    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.291844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.290064    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.348057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.347266    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzvf\" (UniqueName: \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\") pod \"maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.354447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.354203    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:49.459865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.455882    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:49.714086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.714051    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:49.714347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.714317    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:49.900921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.900880    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:06:49.928132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:49.928085    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:50.028831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.022579    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:06:50.028831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.022644    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:06:50.030554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.029925    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:06:50.030554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.030038    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153115    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153185    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153369    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153420    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153546    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153601    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153630    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.153996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153860    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnv4\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.154953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.153928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.250607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.245501    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.250607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.245567    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.265897    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnv4\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.265988    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266034    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266085    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266233    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266289    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266383    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266441    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.266476    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.267268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.267666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.267537    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.268339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.267760    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.269342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.269316    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.269771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.269745    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.322987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.319120    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.350519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.350480    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.389907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.389819    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.391975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.391902    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnv4\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\") pod \"e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") " pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.434817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.434750    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:50.649069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.648613    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:50.649069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.648668    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:50.735977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.729366    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"]
Oct 30 22:06:50.735977 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:50.735758    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea41ad6f_884a_4888_9b62_a42fc00bf653.slice/crio-7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f WatchSource:0}: Error finding container 7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f: Status 404 returned error can't find the container with id 7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f
Oct 30 22:06:50.931163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.931105    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:06:50.967807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.967746    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:50.967911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.967861    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:50.967911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.967905    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:50.975588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:50.975557    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg" event={"ID":"ea41ad6f-884a-4888-9b62-a42fc00bf653","Type":"ContainerStarted","Data":"7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f"}
Oct 30 22:06:51.210914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:51.210744    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.210914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:51.210813    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:51.649428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:51.649390    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 22:06:51.649915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:51.649883    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 22:06:52.022385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.022343    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" event={"ID":"1e31c469-5cab-4c32-83ba-aa5d70741061","Type":"ContainerStarted","Data":"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"}
Oct 30 22:06:52.022543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.022414    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" event={"ID":"1e31c469-5cab-4c32-83ba-aa5d70741061","Type":"ContainerStarted","Data":"8f09b7cef5afac411dc9df28b6d4e0414b46c51d793675ae7130cd923b69a405"}
Oct 30 22:06:52.025000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.024970    5510 generic.go:356] "Generic (PLEG): container finished" podID="ea41ad6f-884a-4888-9b62-a42fc00bf653" containerID="c8d3e8b4cd8ea28101a6c608c5bf969b35ee9dca7d989fa662e0e5f89948d663" exitCode=0
Oct 30 22:06:52.025115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.025049    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg" event={"ID":"ea41ad6f-884a-4888-9b62-a42fc00bf653","Type":"ContainerDied","Data":"c8d3e8b4cd8ea28101a6c608c5bf969b35ee9dca7d989fa662e0e5f89948d663"}
Oct 30 22:06:52.230842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.222898    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.230842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.222981    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:06:52.279803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.268811    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podStartSLOduration=3.268732745 podStartE2EDuration="3.268732745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:52.136879409 +0000 UTC m=+1421.120786287" watchObservedRunningTime="2026-10-30 22:06:52.268732745 +0000 UTC m=+1421.252639624"
Oct 30 22:06:52.435829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.435774    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:52.437123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.437087    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:52.437252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.437147    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:52.917802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.916906    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:06:52.917802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:52.916984    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:53.461014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:53.454586    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:53.461014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:53.454652    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:53.727859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:53.708934    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:06:53.727859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:53.708998    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:54.302806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.294059    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:54.302806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.295035    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:54.335807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.335392    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:06:54.335807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.335464    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:54.373187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.355512    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg" event={"ID":"ea41ad6f-884a-4888-9b62-a42fc00bf653","Type":"ContainerDied","Data":"7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f"}
Oct 30 22:06:54.373187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.355575    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b37a7e9853a1af4cb565a84446207e1fa81cb606e57cef4e17991acb240af3f"
Oct 30 22:06:54.448254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.442857    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:54.448254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.442918    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:54.580599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.531879    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\") pod \"ea41ad6f-884a-4888-9b62-a42fc00bf653\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") "
Oct 30 22:06:54.582436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.582399    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\" (UniqueName: \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\") pod \"ea41ad6f-884a-4888-9b62-a42fc00bf653\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") "
Oct 30 22:06:54.582723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.582694    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\") pod \"ea41ad6f-884a-4888-9b62-a42fc00bf653\" (UID: \"ea41ad6f-884a-4888-9b62-a42fc00bf653\") "
Oct 30 22:06:54.610346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.577672    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca" pod "ea41ad6f-884a-4888-9b62-a42fc00bf653" (UID: "ea41ad6f-884a-4888-9b62-a42fc00bf653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:54.610346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.607403    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:54.610346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.607644    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca" pod "ea41ad6f-884a-4888-9b62-a42fc00bf653" (UID: "ea41ad6f-884a-4888-9b62-a42fc00bf653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:54.610346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.610272    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf" pod "ea41ad6f-884a-4888-9b62-a42fc00bf653" (UID: "ea41ad6f-884a-4888-9b62-a42fc00bf653"). InnerVolumeSpecName "kube-api-access-2mzvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:54.709948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.709775    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzvf\" (UniqueName: \"kubernetes.io/projected/ea41ad6f-884a-4888-9b62-a42fc00bf653-kube-api-access-2mzvf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:54.709948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.709836    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea41ad6f-884a-4888-9b62-a42fc00bf653-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:54.712088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.710506    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:06:54.712088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.710575    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:54.821055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.819179    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:54.821055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.820731    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:54.822408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.822386    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:06:54.874144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.874065    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:54.875177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.875020    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:54.895075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:54.895044    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:06:55.399278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:55.399191    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"
Oct 30 22:06:55.436134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:55.436058    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:55.436425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:55.436363    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:56.435972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.435935    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:56.436429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.436001    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:56.612697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.612657    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"]
Oct 30 22:06:56.613478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.613455    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ea41ad6f-884a-4888-9b62-a42fc00bf653" containerName="revoke-keys"
Oct 30 22:06:56.613478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.613473    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea41ad6f-884a-4888-9b62-a42fc00bf653" containerName="revoke-keys"
Oct 30 22:06:56.613663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.613607    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea41ad6f-884a-4888-9b62-a42fc00bf653" containerName="revoke-keys"
Oct 30 22:06:56.618743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.618724    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.688166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.688084    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"]
Oct 30 22:06:56.712054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.711543    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75v2\" (UniqueName: \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.712054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.711812    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.712342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.712131    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.724947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.724360    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:56.724947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.724394    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:56.724947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.724496    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:56.724947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.724576    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:06:56.813636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.813600    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.813823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.813738    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.813823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.813807    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75v2\" (UniqueName: \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.814542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.814508    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.816090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.816067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.881049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.880957    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75v2\" (UniqueName: \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\") pod \"maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:56.934557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:56.934522    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:57.398232 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:06:57.398197    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cbe0c4_2f56_4d6e_a313_36fd8d02edf3.slice/crio-55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8 WatchSource:0}: Error finding container 55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8: Status 404 returned error can't find the container with id 55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8
Oct 30 22:06:57.400300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:57.400275    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"]
Oct 30 22:06:57.426645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:57.426616    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg" event={"ID":"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3","Type":"ContainerStarted","Data":"55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8"}
Oct 30 22:06:57.436080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:57.435955    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:06:57.437478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:57.436671    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 22:06:58.436743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.436710    5510 generic.go:356] "Generic (PLEG): container finished" podID="a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" containerID="76bc34728dcdd96104ecb421404b0a3bcb3527a500df10e81385a0816ec4bc98" exitCode=0
Oct 30 22:06:58.437253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.436830    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg" event={"ID":"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3","Type":"ContainerDied","Data":"76bc34728dcdd96104ecb421404b0a3bcb3527a500df10e81385a0816ec4bc98"}
Oct 30 22:06:58.438731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.438707    5510 generic.go:356] "Generic (PLEG): container finished" podID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerID="2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc" exitCode=0
Oct 30 22:06:58.438866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.438817    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerDied","Data":"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"}
Oct 30 22:06:58.439346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.439325    5510 scope.go:122] "RemoveContainer" containerID="2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"
Oct 30 22:06:58.440525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.440503    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:58.440704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.440682    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerID="6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd" exitCode=0
Oct 30 22:06:58.440858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.440840    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerDied","Data":"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"}
Oct 30 22:06:58.441047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441027    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:58.441236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441219    5510 scope.go:122] "RemoveContainer" containerID="6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"
Oct 30 22:06:58.441299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441227    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.441342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441305    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.441388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441350    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.441690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.441669    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:06:58.442835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.442808    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.442913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.442868    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.442913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.442902    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:06:58.954827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:58.953464    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.466802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.456602    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerStarted","Data":"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"}
Oct 30 22:06:59.496804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.489017    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerStarted","Data":"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"}
Oct 30 22:06:59.535374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.535343    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:06:59.536732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.536698    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:06:59.536885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.536743    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:06:59.707801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.702657    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:06:59.707801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.702809    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:06:59.707801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.702854    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:06:59.825630 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.825593    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:06:59.925756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.925692    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\") pod \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") "
Oct 30 22:06:59.926010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.925824    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\") pod \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") "
Oct 30 22:06:59.926010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.925923    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\" (UniqueName: \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\") pod \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\" (UID: \"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3\") "
Oct 30 22:06:59.927215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.927185    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca" pod "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" (UID: "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:59.928753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.928724    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca" pod "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" (UID: "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:59.929184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:06:59.929161    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2" pod "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" (UID: "a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3"). InnerVolumeSpecName "kube-api-access-l75v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:00.021861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.021107    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:07:00.029798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.028649    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:00.029798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.028683    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:00.029798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.028701    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75v2\" (UniqueName: \"kubernetes.io/projected/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3-kube-api-access-l75v2\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:00.208810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.151767    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:07:00.421861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.412661    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:00.592806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.589258    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:00.592806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.589327    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:00.592806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.591309    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:00.633802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.594511    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"
Oct 30 22:07:00.633802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.602151    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg" event={"ID":"a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3","Type":"ContainerDied","Data":"55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8"}
Oct 30 22:07:00.633802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.610830    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55fb13709361bacc3109617f81bdbc9cf4ef3bf803f1b828db31659c9cf079e8"
Oct 30 22:07:00.709867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.703474    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:00.709867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:00.703533    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:01.537801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:01.537172    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:01.537801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:01.537235    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:01.707805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:01.699186    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:01.707805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:01.699248    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:02.495804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.489342    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"]
Oct 30 22:07:02.498475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.498443    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" containerName="revoke-keys"
Oct 30 22:07:02.498677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.498657    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" containerName="revoke-keys"
Oct 30 22:07:02.499122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.499102    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" containerName="revoke-keys"
Oct 30 22:07:02.552426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.552390    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:02.552680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.552653    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:02.630869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.630837    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.643918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.643876    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"]
Oct 30 22:07:02.719773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.719730    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.720457    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.720628    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.720949    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.721085    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.721158    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.721668    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.721711    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnt6\" (UniqueName: \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.724147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.721953    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.824393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.824314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.824393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.824349    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnt6\" (UniqueName: \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.824575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.824497    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.824964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.824944    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.827823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.826986    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:02.903929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:02.903600    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnt6\" (UniqueName: \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\") pod \"maas-api-revoke-sub-284b6f16-mclsp\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:03.033963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.033926    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:03.125668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.125638    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.535983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.535945    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:03.536168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.536002    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:03.608837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.608203    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"]
Oct 30 22:07:03.608837 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:07:03.608290    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01bf4dd_4de3_460f_ba49_6efffd96964d.slice/crio-80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2 WatchSource:0}: Error finding container 80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2: Status 404 returned error can't find the container with id 80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2
Oct 30 22:07:03.702973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.702321    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:03.702973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.702386    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:03.727305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.727261    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp" event={"ID":"a01bf4dd-4de3-460f-ba49-6efffd96964d","Type":"ContainerStarted","Data":"80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2"}
Oct 30 22:07:03.844895 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:03.843914    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:07:04.184807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.184369    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:04.535298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.535264    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:04.535465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.535316    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:04.701806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.698195    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:04.701806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.698257    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:04.737660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.737622    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp" event={"ID":"a01bf4dd-4de3-460f-ba49-6efffd96964d","Type":"ContainerStarted","Data":"d420e1c74e8010b7c9888b2adf634fedd9aa6b4d1ee3603f1dcc12716bf8e8eb"}
Oct 30 22:07:04.860416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:04.860266    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp" podStartSLOduration=2.860244458 podStartE2EDuration="2.860244458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:04.847662704 +0000 UTC m=+1433.831569592" watchObservedRunningTime="2026-10-30 22:07:04.860244458 +0000 UTC m=+1433.844151377"
Oct 30 22:07:05.022585 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.022213    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" podUID="90bec354-5da0-4084-a894-982f611629fe" containerName="payload-processing" containerID="cri-o://7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e" gracePeriod=25
Oct 30 22:07:05.185807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.173268    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" podUID="c5826f62-7612-4d90-88a1-daf37f55742f" containerName="payload-pre-processing" containerID="cri-o://25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a" gracePeriod=25
Oct 30 22:07:05.535731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.535344    5510 patch_prober.go:29] interesting pod/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:07:05.535731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.535401    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 30 22:07:05.603666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.596108    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:07:05.640167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.640126    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\") pod \"90bec354-5da0-4084-a894-982f611629fe\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") "
Oct 30 22:07:05.640358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.640211    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\" (UniqueName: \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\") pod \"90bec354-5da0-4084-a894-982f611629fe\" (UID: \"90bec354-5da0-4084-a894-982f611629fe\") "
Oct 30 22:07:05.641905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.641873    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume" pod "90bec354-5da0-4084-a894-982f611629fe" (UID: "90bec354-5da0-4084-a894-982f611629fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:05.645795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.645717    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm" pod "90bec354-5da0-4084-a894-982f611629fe" (UID: "90bec354-5da0-4084-a894-982f611629fe"). InnerVolumeSpecName "kube-api-access-rn5hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:05.698402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.698364    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:07:05.698574 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.698432    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:07:05.713295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.713265    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:07:05.744722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.742232    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\") pod \"c5826f62-7612-4d90-88a1-daf37f55742f\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") "
Oct 30 22:07:05.744722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.742321    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\" (UniqueName: \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\") pod \"c5826f62-7612-4d90-88a1-daf37f55742f\" (UID: \"c5826f62-7612-4d90-88a1-daf37f55742f\") "
Oct 30 22:07:05.744722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.743520    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bec354-5da0-4084-a894-982f611629fe-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:05.744722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.743548    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn5hm\" (UniqueName: \"kubernetes.io/projected/90bec354-5da0-4084-a894-982f611629fe-kube-api-access-rn5hm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:05.744722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.744263    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume" pod "c5826f62-7612-4d90-88a1-daf37f55742f" (UID: "c5826f62-7612-4d90-88a1-daf37f55742f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:05.754968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.754930    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz" pod "c5826f62-7612-4d90-88a1-daf37f55742f" (UID: "c5826f62-7612-4d90-88a1-daf37f55742f"). InnerVolumeSpecName "kube-api-access-2mwlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:05.767171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.767139    5510 generic.go:356] "Generic (PLEG): container finished" podID="c5826f62-7612-4d90-88a1-daf37f55742f" containerID="25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a" exitCode=0
Oct 30 22:07:05.767332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.767297    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" event={"ID":"c5826f62-7612-4d90-88a1-daf37f55742f","Type":"ContainerDied","Data":"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"}
Oct 30 22:07:05.767429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.767362    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl" event={"ID":"c5826f62-7612-4d90-88a1-daf37f55742f","Type":"ContainerDied","Data":"d7ece68c1bcd8d20bcba7241b450a36006a0c531537c25ece79725b0aba176a1"}
Oct 30 22:07:05.767429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.767408    5510 scope.go:122] "RemoveContainer" containerID="25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"
Oct 30 22:07:05.767620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.767604    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"
Oct 30 22:07:05.773396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.773366    5510 generic.go:356] "Generic (PLEG): container finished" podID="a01bf4dd-4de3-460f-ba49-6efffd96964d" containerID="d420e1c74e8010b7c9888b2adf634fedd9aa6b4d1ee3603f1dcc12716bf8e8eb" exitCode=0
Oct 30 22:07:05.773626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.773513    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp" event={"ID":"a01bf4dd-4de3-460f-ba49-6efffd96964d","Type":"ContainerDied","Data":"d420e1c74e8010b7c9888b2adf634fedd9aa6b4d1ee3603f1dcc12716bf8e8eb"}
Oct 30 22:07:05.775877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.775806    5510 generic.go:356] "Generic (PLEG): container finished" podID="90bec354-5da0-4084-a894-982f611629fe" containerID="7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e" exitCode=0
Oct 30 22:07:05.775975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.775894    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" event={"ID":"90bec354-5da0-4084-a894-982f611629fe","Type":"ContainerDied","Data":"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"}
Oct 30 22:07:05.775975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.775937    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5" event={"ID":"90bec354-5da0-4084-a894-982f611629fe","Type":"ContainerDied","Data":"bc68d0a8f0f16c44847f0aac42f325c4cfa5cdb1077c37e7bf8b6e50f3979443"}
Oct 30 22:07:05.776090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.776000    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"
Oct 30 22:07:05.846274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.846244    5510 scope.go:122] "RemoveContainer" containerID="25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"
Oct 30 22:07:05.847694 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:05.847659    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a\": container with ID starting with 25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a not found: ID does not exist" containerID="25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"
Oct 30 22:07:05.847871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.847707    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a"} err="failed to get container status \"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a\": rpc error: code = NotFound desc = could not find container \"25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a\": container with ID starting with 25815ce23f808c551c5dd9e0223789c2a59f62c532a9f0c7d3c3ae7841fee94a not found: ID does not exist"
Oct 30 22:07:05.847871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.847730    5510 scope.go:122] "RemoveContainer" containerID="7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"
Oct 30 22:07:05.850963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.850940    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5826f62-7612-4d90-88a1-daf37f55742f-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:05.851110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.851093    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwlz\" (UniqueName: \"kubernetes.io/projected/c5826f62-7612-4d90-88a1-daf37f55742f-kube-api-access-2mwlz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:05.867475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.867144    5510 scope.go:122] "RemoveContainer" containerID="7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"
Oct 30 22:07:05.867791 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:05.867672    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e\": container with ID starting with 7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e not found: ID does not exist" containerID="7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"
Oct 30 22:07:05.867791 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:05.867712    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e"} err="failed to get container status \"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e\": rpc error: code = NotFound desc = could not find container \"7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e\": container with ID starting with 7d60816b39f719ccc05051b3605db1eb0db516a5ddbcf614fc25f9d47f2f984e not found: ID does not exist"
Oct 30 22:07:06.063806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.059760    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:07:06.066019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.065991    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3bf5aa7a-6596f4cfcf-tn7dl"]
Oct 30 22:07:06.162342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.162312    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:07:06.177611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.177567    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3bf5aa7a-56c684d6d6-7z6f5"]
Oct 30 22:07:06.539655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.539623    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:07:06.540963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.540938    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:07:06.702136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.702102    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:07:06.702529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.702508    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:07:06.703148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:06.703127    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:07:07.032344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.032314    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:07.073499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.073404    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\" (UniqueName: \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\") pod \"a01bf4dd-4de3-460f-ba49-6efffd96964d\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") "
Oct 30 22:07:07.073499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.073493    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\") pod \"a01bf4dd-4de3-460f-ba49-6efffd96964d\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") "
Oct 30 22:07:07.073725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.073553    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\") pod \"a01bf4dd-4de3-460f-ba49-6efffd96964d\" (UID: \"a01bf4dd-4de3-460f-ba49-6efffd96964d\") "
Oct 30 22:07:07.073892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.073867    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca" pod "a01bf4dd-4de3-460f-ba49-6efffd96964d" (UID: "a01bf4dd-4de3-460f-ba49-6efffd96964d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:07.074434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.074413    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.075610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.075584    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca" pod "a01bf4dd-4de3-460f-ba49-6efffd96964d" (UID: "a01bf4dd-4de3-460f-ba49-6efffd96964d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:07.075725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.075704    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6" pod "a01bf4dd-4de3-460f-ba49-6efffd96964d" (UID: "a01bf4dd-4de3-460f-ba49-6efffd96964d"). InnerVolumeSpecName "kube-api-access-mhnt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:07.128037 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.127998    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bec354-5da0-4084-a894-982f611629fe" path="/var/lib/kubelet/pods/90bec354-5da0-4084-a894-982f611629fe/volumes"
Oct 30 22:07:07.128512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.128498    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5826f62-7612-4d90-88a1-daf37f55742f" path="/var/lib/kubelet/pods/c5826f62-7612-4d90-88a1-daf37f55742f/volumes"
Oct 30 22:07:07.185222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.180685    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01bf4dd-4de3-460f-ba49-6efffd96964d-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.185222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.180722    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnt6\" (UniqueName: \"kubernetes.io/projected/a01bf4dd-4de3-460f-ba49-6efffd96964d-kube-api-access-mhnt6\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.584855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.584774    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:07.867462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.866515    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"
Oct 30 22:07:07.870791 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.870724    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp" event={"ID":"a01bf4dd-4de3-460f-ba49-6efffd96964d","Type":"ContainerDied","Data":"80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2"}
Oct 30 22:07:07.870949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:07.870818    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80df37c4580bccf41eaf00f6e693832f70e80e39ed0e9579eef88f74993281b2"
Oct 30 22:07:08.103799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:08.099956    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:07:08.268802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:08.267805    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:07:08.544261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:08.544180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:08.770325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:08.769656    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:09.035032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.034969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:09.290818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.290717    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:09.533351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.533305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:09.576436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.576335    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"]
Oct 30 22:07:09.623032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.622997    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3084d7f-wxhlf"]
Oct 30 22:07:09.725705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:09.725674    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:10.437815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:10.437660    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:10.590808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:10.588775    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:10.774803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:10.763635    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:11.301803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.284514    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dc6b43-deb9-428e-bf8b-065b276d690c" path="/var/lib/kubelet/pods/31dc6b43-deb9-428e-bf8b-065b276d690c/volumes"
Oct 30 22:07:11.952660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.952628    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:07:11.953471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953453    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a01bf4dd-4de3-460f-ba49-6efffd96964d" containerName="revoke-keys"
Oct 30 22:07:11.953535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953473    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01bf4dd-4de3-460f-ba49-6efffd96964d" containerName="revoke-keys"
Oct 30 22:07:11.953535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953487    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c5826f62-7612-4d90-88a1-daf37f55742f" containerName="payload-pre-processing"
Oct 30 22:07:11.953535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953493    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5826f62-7612-4d90-88a1-daf37f55742f" containerName="payload-pre-processing"
Oct 30 22:07:11.953535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953507    5510 cpu_manager.go:414] "containerMap: removing container" podUID="90bec354-5da0-4084-a894-982f611629fe" containerName="payload-processing"
Oct 30 22:07:11.953535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953512    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bec354-5da0-4084-a894-982f611629fe" containerName="payload-processing"
Oct 30 22:07:11.953710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953628    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bec354-5da0-4084-a894-982f611629fe" containerName="payload-processing"
Oct 30 22:07:11.953710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953647    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01bf4dd-4de3-460f-ba49-6efffd96964d" containerName="revoke-keys"
Oct 30 22:07:11.953710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.953655    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5826f62-7612-4d90-88a1-daf37f55742f" containerName="payload-pre-processing"
Oct 30 22:07:11.958739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:11.958722    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.009639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.009607    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-13297892-gw-openshift-default-dockercfg-hlfqp\""
Oct 30 22:07:12.068833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.068457    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:07:12.072142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.072109    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:12.226741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.222823    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.226741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.222923    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.226741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.222954    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.228002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.227846    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhbf\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.228002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.227903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.228002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.227962    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.228256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.228033    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.228256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.228181    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.240191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.236016    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.337522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.337485    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhbf\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.337799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.337758    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.337968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.337949    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.338140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.338126    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.338347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.338327    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.338541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.338523    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.338764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.338743    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.338989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.338969    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.339207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.339184    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.339405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.339259    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.339695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.339654    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.341838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.341094    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.341838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.341331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.341961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.341842    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.343905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.343877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.347304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.347279    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.396700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.396660    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:12.404814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.398846    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.409838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.405964    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhbf\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\") pod \"e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") " pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:12.577693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:12.577528    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:13.026130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.026095    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:07:13.030724 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:07:13.030689    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2b9342_f6e2_4150_a585_e7db7b2a9a64.slice/crio-d9317e1c6b26f4c948d5b90339d22f106d605fc810769eb857c43035af22adf4 WatchSource:0}: Error finding container d9317e1c6b26f4c948d5b90339d22f106d605fc810769eb857c43035af22adf4: Status 404 returned error can't find the container with id d9317e1c6b26f4c948d5b90339d22f106d605fc810769eb857c43035af22adf4
Oct 30 22:07:13.034011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.033947    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:13.034120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.034041    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:13.034120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.034082    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:13.100645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.100548    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" podUID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" containerName="payload-processing" containerID="cri-o://98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0" gracePeriod=25
Oct 30 22:07:13.268459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.268344    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" podUID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" containerName="payload-pre-processing" containerID="cri-o://7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8" gracePeriod=25
Oct 30 22:07:13.312890 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:13.312082    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:07:13.426644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.426615    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:13.432680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.432653    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:07:13.583889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.582033    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\") pod \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") "
Oct 30 22:07:13.583889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.582094    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\" (UniqueName: \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\") pod \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\" (UID: \"ca7ce8ef-8c22-4f13-8a13-8eb10956720e\") "
Oct 30 22:07:13.584223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.584196    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume" pod "ca7ce8ef-8c22-4f13-8a13-8eb10956720e" (UID: "ca7ce8ef-8c22-4f13-8a13-8eb10956720e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:13.587291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.587251    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79" pod "ca7ce8ef-8c22-4f13-8a13-8eb10956720e" (UID: "ca7ce8ef-8c22-4f13-8a13-8eb10956720e"). InnerVolumeSpecName "kube-api-access-w6x79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:13.593667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.593340    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:07:13.683929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.683886    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\" (UniqueName: \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\") pod \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") "
Oct 30 22:07:13.684171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.684155    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\") pod \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\" (UID: \"c33287dd-ab87-4985-ac1e-9aa1a8c02c39\") "
Oct 30 22:07:13.684511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.684485    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume" pod "c33287dd-ab87-4985-ac1e-9aa1a8c02c39" (UID: "c33287dd-ab87-4985-ac1e-9aa1a8c02c39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:13.685047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.685015    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:13.685155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.685049    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6x79\" (UniqueName: \"kubernetes.io/projected/ca7ce8ef-8c22-4f13-8a13-8eb10956720e-kube-api-access-w6x79\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:13.685155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.685066    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:13.686379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.686352    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp" pod "c33287dd-ab87-4985-ac1e-9aa1a8c02c39" (UID: "c33287dd-ab87-4985-ac1e-9aa1a8c02c39"). InnerVolumeSpecName "kube-api-access-42psp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:13.786351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.786258    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42psp\" (UniqueName: \"kubernetes.io/projected/c33287dd-ab87-4985-ac1e-9aa1a8c02c39-kube-api-access-42psp\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:13.965298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.965267    5510 generic.go:356] "Generic (PLEG): container finished" podID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" containerID="7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8" exitCode=0
Oct 30 22:07:13.965507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.965328    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"
Oct 30 22:07:13.965507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.965357    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" event={"ID":"c33287dd-ab87-4985-ac1e-9aa1a8c02c39","Type":"ContainerDied","Data":"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"}
Oct 30 22:07:13.965507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.965427    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w" event={"ID":"c33287dd-ab87-4985-ac1e-9aa1a8c02c39","Type":"ContainerDied","Data":"bd5674e7f35d23ed518067db8b6eee6d9f1e3a151d1a243926b99dcabb8f66e7"}
Oct 30 22:07:13.965507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.965458    5510 scope.go:122] "RemoveContainer" containerID="7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"
Oct 30 22:07:13.967847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.967740    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerStarted","Data":"9356071a480c1740cfe04edc3bb11bf39f162c9b67c4a9184a356df492163a23"}
Oct 30 22:07:13.968055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.968027    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerStarted","Data":"d9317e1c6b26f4c948d5b90339d22f106d605fc810769eb857c43035af22adf4"}
Oct 30 22:07:13.969393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.969370    5510 generic.go:356] "Generic (PLEG): container finished" podID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" containerID="98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0" exitCode=0
Oct 30 22:07:13.969510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.969403    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"
Oct 30 22:07:13.969510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.969452    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" event={"ID":"ca7ce8ef-8c22-4f13-8a13-8eb10956720e","Type":"ContainerDied","Data":"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"}
Oct 30 22:07:13.969510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.969490    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz" event={"ID":"ca7ce8ef-8c22-4f13-8a13-8eb10956720e","Type":"ContainerDied","Data":"4e53e0cadcaecca8f895704b0171adf7aef87905e1654a198feff6e41264f5cd"}
Oct 30 22:07:13.979858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.979842    5510 scope.go:122] "RemoveContainer" containerID="7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"
Oct 30 22:07:13.980118 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:13.980098    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8\": container with ID starting with 7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8 not found: ID does not exist" containerID="7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"
Oct 30 22:07:13.980216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.980123    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8"} err="failed to get container status \"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8\": rpc error: code = NotFound desc = could not find container \"7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8\": container with ID starting with 7cf9b698a2508cb0d9c49f4c066a6ada7a16f6202f83f50ba0c0a286514c4ec8 not found: ID does not exist"
Oct 30 22:07:13.980216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.980137    5510 scope.go:122] "RemoveContainer" containerID="98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"
Oct 30 22:07:13.990959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.990940    5510 scope.go:122] "RemoveContainer" containerID="98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"
Oct 30 22:07:13.991264 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:13.991245    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0\": container with ID starting with 98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0 not found: ID does not exist" containerID="98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"
Oct 30 22:07:13.991325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:13.991270    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0"} err="failed to get container status \"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0\": rpc error: code = NotFound desc = could not find container \"98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0\": container with ID starting with 98ec917df0c7e221b361a2fb8edb7d28acf4c8d65ead275d6923cb8717729ee0 not found: ID does not exist"
Oct 30 22:07:14.120241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.120157    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:07:14.164016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.163906    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:14.168744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.168709    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ff7514-77c4b94fc9-2df9w"]
Oct 30 22:07:14.235518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.235281    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podStartSLOduration=3.235254457 podStartE2EDuration="3.235254457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:14.23038504 +0000 UTC m=+1443.214291916" watchObservedRunningTime="2026-10-30 22:07:14.235254457 +0000 UTC m=+1443.219161332"
Oct 30 22:07:14.340827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.340740    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:07:14.376174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.375632    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:14.415826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.415635    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50ff7514-5c78469874-8w6rz"]
Oct 30 22:07:14.578753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.578713    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:14.584700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.584677    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:14.982863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.982819    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:14.983919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:14.983898    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:07:15.130393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:15.130360    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" path="/var/lib/kubelet/pods/c33287dd-ab87-4985-ac1e-9aa1a8c02c39/volumes"
Oct 30 22:07:15.131318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:15.130963    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" path="/var/lib/kubelet/pods/ca7ce8ef-8c22-4f13-8a13-8eb10956720e/volumes"
Oct 30 22:07:16.634356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:16.634320    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.832401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:16.832369    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:18.035725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:18.035689    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:07:19.097016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:19.096972    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:19.344048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:19.343994    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.077473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:20.077444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:21.061541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.061462    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"]
Oct 30 22:07:21.104397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.104361    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"]
Oct 30 22:07:21.109095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.109067    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7fc7cab-fp788"]
Oct 30 22:07:21.113612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.113583    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3588cb87-bdb7x"]
Oct 30 22:07:21.128817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.128754    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f637f6e-b239-4271-9822-fbdea237ccf5" path="/var/lib/kubelet/pods/7f637f6e-b239-4271-9822-fbdea237ccf5/volumes"
Oct 30 22:07:21.129302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:21.129287    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a314676d-b945-4a06-9b97-6a8907b0db2e" path="/var/lib/kubelet/pods/a314676d-b945-4a06-9b97-6a8907b0db2e/volumes"
Oct 30 22:07:24.281803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:24.277910    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:25.046349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:25.046317    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:28.307130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:28.307094    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:29.055457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:29.055419    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"]
Oct 30 22:07:29.199865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:29.199832    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9b6907-5t4mp"]
Oct 30 22:07:29.786660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:29.786624    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.018167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.018128    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.205868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.205014    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.366875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.366839    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"]
Oct 30 22:07:30.420907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.420870    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3bf5aa7a-p5hrg"]
Oct 30 22:07:30.838884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.838848    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:07:30.839393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.839106    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy" containerID="cri-o://f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1" gracePeriod=30
Oct 30 22:07:30.844672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:30.844646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:31.383853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:31.380231    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a439047-3880-4a10-ae86-71721fd634ee" path="/var/lib/kubelet/pods/5a439047-3880-4a10-ae86-71721fd634ee/volumes"
Oct 30 22:07:31.383853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:31.380873    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea41ad6f-884a-4888-9b62-a42fc00bf653" path="/var/lib/kubelet/pods/ea41ad6f-884a-4888-9b62-a42fc00bf653/volumes"
Oct 30 22:07:31.533203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:31.533166    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.079811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.079091    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:07:32.080405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080378    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" containerName="payload-processing"
Oct 30 22:07:32.080491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080409    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" containerName="payload-processing"
Oct 30 22:07:32.080491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080432    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" containerName="payload-pre-processing"
Oct 30 22:07:32.080491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080441    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" containerName="payload-pre-processing"
Oct 30 22:07:32.080691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080670    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca7ce8ef-8c22-4f13-8a13-8eb10956720e" containerName="payload-processing"
Oct 30 22:07:32.080752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.080697    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33287dd-ab87-4985-ac1e-9aa1a8c02c39" containerName="payload-pre-processing"
Oct 30 22:07:32.111993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.111961    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.164127 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.164003    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-de9c059f-openshift-default-dockercfg-zhbhd\""
Oct 30 22:07:32.217800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.200038    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztwk\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.218262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.218222    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.218544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.218524    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.226015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.225970    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.252078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.252038    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.252340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.252322    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.252504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.252486    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.252638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.252623    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.252768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.252755    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359575    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359654    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359697    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359765    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359829    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359871    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.359967    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztwk\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.360088    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.371907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.360207    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.393801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.375335    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.393801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.376910    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.393801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.390861    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.393801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.391166    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.393801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.391429    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.467806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.453550    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.473437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.468877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.505102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.505065    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:07:32.574653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.556589    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.622771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.622090    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.672263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.672226    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztwk\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.681842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.677193    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\") pod \"e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") " pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:32.788917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:32.788597    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:33.346846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.346809    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:07:33.352802 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:07:33.352760    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf151195_cc3c_44b1_a8e3_b6487015d1f1.slice/crio-2fc8330025854c79bdaaaaf433f8bcd2155eb48e4019638cb81e63acf16a428f WatchSource:0}: Error finding container 2fc8330025854c79bdaaaaf433f8bcd2155eb48e4019638cb81e63acf16a428f: Status 404 returned error can't find the container with id 2fc8330025854c79bdaaaaf433f8bcd2155eb48e4019638cb81e63acf16a428f
Oct 30 22:07:33.355593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.355559    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:33.355702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.355641    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:33.355764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.355709    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:07:33.462027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.461983    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerStarted","Data":"510132f7fd014b77c2c92fba747d84ca404fa3be52d4111a1baa89748274c963"}
Oct 30 22:07:33.462202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.462048    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerStarted","Data":"2fc8330025854c79bdaaaaf433f8bcd2155eb48e4019638cb81e63acf16a428f"}
Oct 30 22:07:33.580713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.580656    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podStartSLOduration=2.580640967 podStartE2EDuration="2.580640967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:33.576856764 +0000 UTC m=+1462.560763640" watchObservedRunningTime="2026-10-30 22:07:33.580640967 +0000 UTC m=+1462.564547842"
Oct 30 22:07:33.790802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:33.789899    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:34.371667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:34.371623    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:34.795573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:34.795540    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:35.646194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:35.646151    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:35.646980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:35.646958    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:07:35.725030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:35.725001    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:35.885002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:35.884975    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.056632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.051616    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.264466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.244495    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.290856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.282333    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:07:36.366513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366477    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366708    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366729    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366756    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366795    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.366903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366767    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:36.367131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366957    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.367131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.366977    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.367131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.367009    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\") pod \"5bd66532-5265-43f5-a3d6-8a5db541364b\" (UID: \"5bd66532-5265-43f5-a3d6-8a5db541364b\") "
Oct 30 22:07:36.367652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.367626    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.368047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.368020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:36.368142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.368116    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:36.368199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.368146    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:36.368377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.368357    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:36.369487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.369440    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:36.370275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.370252    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "kube-api-access-kkzls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:36.370510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.370490    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:36.370765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.370744    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy" pod "5bd66532-5265-43f5-a3d6-8a5db541364b" (UID: "5bd66532-5265-43f5-a3d6-8a5db541364b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:36.465799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.465752    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.469855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469826    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.469855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469858    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469873    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469887    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzls\" (UniqueName: \"kubernetes.io/projected/5bd66532-5265-43f5-a3d6-8a5db541364b-kube-api-access-kkzls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469901    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469914    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469927    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd66532-5265-43f5-a3d6-8a5db541364b-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.470045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.469940    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd66532-5265-43f5-a3d6-8a5db541364b-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.657509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.657415    5510 generic.go:356] "Generic (PLEG): container finished" podID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerID="f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1" exitCode=0
Oct 30 22:07:36.657509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.657465    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"
Oct 30 22:07:36.658046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.657508    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerDied","Data":"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"}
Oct 30 22:07:36.658046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.657572    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg" event={"ID":"5bd66532-5265-43f5-a3d6-8a5db541364b","Type":"ContainerDied","Data":"57a4681fe4620ee9afc8447b27a738993dd4e42f1be45eea8964debf814cd594"}
Oct 30 22:07:36.658046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.657602    5510 scope.go:122] "RemoveContainer" containerID="f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"
Oct 30 22:07:36.677806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.677765    5510 scope.go:122] "RemoveContainer" containerID="6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"
Oct 30 22:07:36.689609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.689588    5510 scope.go:122] "RemoveContainer" containerID="f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"
Oct 30 22:07:36.689928 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:36.689906    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1\": container with ID starting with f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1 not found: ID does not exist" containerID="f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"
Oct 30 22:07:36.690016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.689939    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1"} err="failed to get container status \"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1\": rpc error: code = NotFound desc = could not find container \"f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1\": container with ID starting with f1e56a0fb4938997d80e9369a5a6b093514ed2df84756b4b7fbf6bde3a294ea1 not found: ID does not exist"
Oct 30 22:07:36.690016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.689957    5510 scope.go:122] "RemoveContainer" containerID="6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"
Oct 30 22:07:36.690242 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:36.690224    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd\": container with ID starting with 6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd not found: ID does not exist" containerID="6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"
Oct 30 22:07:36.690302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.690246    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd"} err="failed to get container status \"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd\": rpc error: code = NotFound desc = could not find container \"6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd\": container with ID starting with 6e66eb4d218349649b4eda9d8695d7b955bbb653990fb56978343ac600669bcd not found: ID does not exist"
Oct 30 22:07:36.747677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.747622    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:36.843804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.839846    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:07:36.850520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.850486    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3bf5aa7a-openshift-default-6f55f6b889-x6vtg"]
Oct 30 22:07:36.936398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.936304    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"]
Oct 30 22:07:36.942109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:36.942085    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ff7514-qh6pg"]
Oct 30 22:07:37.127221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:37.127189    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" path="/var/lib/kubelet/pods/5bd66532-5265-43f5-a3d6-8a5db541364b/volumes"
Oct 30 22:07:37.127710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:37.127695    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3" path="/var/lib/kubelet/pods/a7cbe0c4-2f56-4d6e-a313-36fd8d02edf3/volumes"
Oct 30 22:07:38.050373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.050341    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"]
Oct 30 22:07:38.098343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.098307    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebb856e0-wg6fl"]
Oct 30 22:07:38.559718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.559682    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:07:38.560014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.559982    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" containerID="cri-o://bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd" gracePeriod=30
Oct 30 22:07:38.566194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.566162    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:07:38.566466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.566385    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy" containerID="cri-o://56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc" gracePeriod=30
Oct 30 22:07:38.590501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.590475    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:38.883072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:38.882934    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:39.052632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.045261    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:39.454804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.437689    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219f9b2f-3ad8-4172-8bea-d032ce6680a9" path="/var/lib/kubelet/pods/219f9b2f-3ad8-4172-8bea-d032ce6680a9/volumes"
Oct 30 22:07:39.492621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.492584    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:07:39.548870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.547906    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:07:39.548870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.547946    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:07:39.548870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.548284    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:07:39.630900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.619551    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.630900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.630061    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:07:39.631854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.631820    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:39.676276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.669225    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6wq\" (UniqueName: \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.680590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.676856    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.681208 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.679751    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-de9c059f-dockercfg-7j42f\""
Oct 30 22:07:39.681383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.679868    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-de9c059f\""
Oct 30 22:07:39.804805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.795166    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.804805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.795586    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6wq\" (UniqueName: \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.845612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.845572    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:39.928018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.927984    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6wq\" (UniqueName: \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\") pod \"payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") " pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:40.004862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.985286    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:40.013507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:39.994945    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:40.405378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:40.405282    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:07:40.411579 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:07:40.411549    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7733fa_df76_4920_91c6_a92b58bfa294.slice/crio-a81469f275d1ab382bfdf073a4757d263260e9767b3f7d802fcb08ab5ec89e01 WatchSource:0}: Error finding container a81469f275d1ab382bfdf073a4757d263260e9767b3f7d802fcb08ab5ec89e01: Status 404 returned error can't find the container with id a81469f275d1ab382bfdf073a4757d263260e9767b3f7d802fcb08ab5ec89e01
Oct 30 22:07:40.601084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:40.601052    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:40.900760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:40.900718    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" event={"ID":"dd7733fa-df76-4920-91c6-a92b58bfa294","Type":"ContainerStarted","Data":"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"}
Oct 30 22:07:40.900999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:40.900795    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" event={"ID":"dd7733fa-df76-4920-91c6-a92b58bfa294","Type":"ContainerStarted","Data":"a81469f275d1ab382bfdf073a4757d263260e9767b3f7d802fcb08ab5ec89e01"}
Oct 30 22:07:40.901068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:40.901007    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:41.008919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:41.008864    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" podStartSLOduration=2.008846843 podStartE2EDuration="2.008846843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:41.007433491 +0000 UTC m=+1469.991340368" watchObservedRunningTime="2026-10-30 22:07:41.008846843 +0000 UTC m=+1469.992753722"
Oct 30 22:07:41.263477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:41.263359    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:43.285117 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:43.285075    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:07:43.443218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.443183    5510 patch_prober.go:29] interesting pod/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:43.443375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.443234    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:43.866709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.866679    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:07:43.926675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.926642    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:07:43.927073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.927053    5510 generic.go:356] "Generic (PLEG): container finished" podID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerID="56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc" exitCode=0
Oct 30 22:07:43.927133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.927089    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerDied","Data":"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"}
Oct 30 22:07:43.927168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.927146    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc" event={"ID":"d7b515f9-9ff1-4884-b543-4c138abc4258","Type":"ContainerDied","Data":"ad6999e056255fd9528d2e7261af7b7f4fb668bd9d27567543fbd8bd5c850b8a"}
Oct 30 22:07:43.927212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.927169    5510 scope.go:122] "RemoveContainer" containerID="56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"
Oct 30 22:07:43.928729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.928705    5510 generic.go:356] "Generic (PLEG): container finished" podID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerID="bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd" exitCode=0
Oct 30 22:07:43.928869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.928747    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"
Oct 30 22:07:43.928869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.928791    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" event={"ID":"1e31c469-5cab-4c32-83ba-aa5d70741061","Type":"ContainerDied","Data":"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"}
Oct 30 22:07:43.928869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.928846    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz" event={"ID":"1e31c469-5cab-4c32-83ba-aa5d70741061","Type":"ContainerDied","Data":"8f09b7cef5afac411dc9df28b6d4e0414b46c51d793675ae7130cd923b69a405"}
Oct 30 22:07:43.938133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.938115    5510 scope.go:122] "RemoveContainer" containerID="2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.951151    5510 scope.go:122] "RemoveContainer" containerID="56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:43.951540    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc\": container with ID starting with 56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc not found: ID does not exist" containerID="56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.951573    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc"} err="failed to get container status \"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc\": rpc error: code = NotFound desc = could not find container \"56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc\": container with ID starting with 56096a90ada55e76068cfdee9f290206dff5d99d3e9ed83ff0e4e04faa419ccc not found: ID does not exist"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.951594    5510 scope.go:122] "RemoveContainer" containerID="2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:43.951908    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc\": container with ID starting with 2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc not found: ID does not exist" containerID="2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.951934    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc"} err="failed to get container status \"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc\": rpc error: code = NotFound desc = could not find container \"2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc\": container with ID starting with 2cbb942770b3ec9339ae90204dd6ddb4967c31b582a6e18b5f9984a0bceb61bc not found: ID does not exist"
Oct 30 22:07:43.952105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.951954    5510 scope.go:122] "RemoveContainer" containerID="bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"
Oct 30 22:07:43.962468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.962453    5510 scope.go:122] "RemoveContainer" containerID="bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"
Oct 30 22:07:43.962754 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:07:43.962734    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd\": container with ID starting with bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd not found: ID does not exist" containerID="bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"
Oct 30 22:07:43.962837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:43.962764    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd"} err="failed to get container status \"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd\": rpc error: code = NotFound desc = could not find container \"bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd\": container with ID starting with bf31aa44d32f17f3b4823e5ef70a0fdaea76ac83e3b7835d91d1a7ab7eecf7cd not found: ID does not exist"
Oct 30 22:07:44.014416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014381    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014446    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014475    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.014586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014498    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.014586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014557    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014592    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014617    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014642    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014673    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014699    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014733    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.014841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014752    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014841    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014909    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014972    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.014978    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.015058    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.015205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.015190    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.015533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.015206    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\") pod \"1e31c469-5cab-4c32-83ba-aa5d70741061\" (UID: \"1e31c469-5cab-4c32-83ba-aa5d70741061\") "
Oct 30 22:07:44.015533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.015316    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\") pod \"d7b515f9-9ff1-4884-b543-4c138abc4258\" (UID: \"d7b515f9-9ff1-4884-b543-4c138abc4258\") "
Oct 30 22:07:44.015533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.015471    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016038    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016380    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016404    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b515f9-9ff1-4884-b543-4c138abc4258-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016418    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016432    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.016928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016654    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.017312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.016986    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.018352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.018325    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.018629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.018605    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:44.018867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.018842    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.019019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.019002    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.019296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.019274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.020620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.020592    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.020738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.020716    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.020880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.020793    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:44.020971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.020947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:44.021051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.021020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75" pod "d7b515f9-9ff1-4884-b543-4c138abc4258" (UID: "d7b515f9-9ff1-4884-b543-4c138abc4258"). InnerVolumeSpecName "kube-api-access-mfg75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.021360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.021336    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "kube-api-access-zpnv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.022102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.022074    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo" pod "1e31c469-5cab-4c32-83ba-aa5d70741061" (UID: "1e31c469-5cab-4c32-83ba-aa5d70741061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118484    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118513    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118525    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118534    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118543    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118552    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118560    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118569    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e31c469-5cab-4c32-83ba-aa5d70741061-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118582    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e31c469-5cab-4c32-83ba-aa5d70741061-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118595    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfg75\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-kube-api-access-mfg75\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118605    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b515f9-9ff1-4884-b543-4c138abc4258-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118613    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b515f9-9ff1-4884-b543-4c138abc4258-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118623    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e31c469-5cab-4c32-83ba-aa5d70741061-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.118956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.118632    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnv4\" (UniqueName: \"kubernetes.io/projected/1e31c469-5cab-4c32-83ba-aa5d70741061-kube-api-access-zpnv4\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.344427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.344393    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:07:44.350606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.350577    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-k9wqz"]
Oct 30 22:07:44.939416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:44.939335    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"
Oct 30 22:07:45.097053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:45.097019    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:07:45.127477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:45.127444    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" path="/var/lib/kubelet/pods/1e31c469-5cab-4c32-83ba-aa5d70741061/volumes"
Oct 30 22:07:45.142541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:45.142510    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50ff7514-openshift-default-679cf6446f-mf6cc"]
Oct 30 22:07:45.919355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:45.919316    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:47.127705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:47.127671    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" path="/var/lib/kubelet/pods/d7b515f9-9ff1-4884-b543-4c138abc4258/volumes"
Oct 30 22:07:50.359809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:50.356886    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:51.911721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:51.911690    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:07:54.730862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:54.730831    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:56.971900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:56.970937    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 22:07:56.971900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:56.971201    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" containerID="cri-o://83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e" gracePeriod=30
Oct 30 22:07:57.006909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:57.006847    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:07:57.007332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:57.007301    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy" containerID="cri-o://ee2b2039225677ab6b38574840773e9b24445a0a985a0aa283011e49bb9d020c" gracePeriod=30
Oct 30 22:07:57.060883 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:57.053485    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:07:57.072403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:57.072353    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" containerID="cri-o://d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de" gracePeriod=30
Oct 30 22:07:58.058841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.052894    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:58.058841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.053349    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:07:58.058841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.053683    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:07:58.188805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.188737    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:58.189517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.189487    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:07:58.222512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:58.222473    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:59.275608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:59.275573    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:59.443264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:59.443201    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:59.604408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:59.604302    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:59.624975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:59.624937    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:59.625249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:07:59.625222    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:01.973804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:01.964887    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:08:01.973804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:01.964978    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:08:02.373211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.373177    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 22:08:02.434080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.434054    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:08:02.563151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563115    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563176    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563251    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563299    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563353    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.563593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563411    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563501    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563539    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.563593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563570    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563594    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563625    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563706    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563734    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563853    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.563891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563859    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.564185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563931    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.564185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.563978    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\") pod \"2793053a-78e8-43d1-87e3-9f753ecd6e89\" (UID: \"2793053a-78e8-43d1-87e3-9f753ecd6e89\") "
Oct 30 22:08:02.564185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.564054    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.564185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.564096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.564399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.564232    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.564516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.564495    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\") pod \"20563428-9ca2-4c60-ab11-fb4930f95a61\" (UID: \"20563428-9ca2-4c60-ab11-fb4930f95a61\") "
Oct 30 22:08:02.565338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.564607    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.565338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.565054    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.565594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.565434    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.565921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.565894    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.566036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566014    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2793053a-78e8-43d1-87e3-9f753ecd6e89-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566037    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566051    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566065    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566079    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566093    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.566411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566218    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.566411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566261    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.566543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566452    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.566824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.566770    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:02.567469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.567439    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:02.569533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.569487    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.569533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.569499    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:02.569533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.569508    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "kube-api-access-tzn9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.569672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.569537    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.569768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.569746    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token" pod "2793053a-78e8-43d1-87e3-9f753ecd6e89" (UID: "2793053a-78e8-43d1-87e3-9f753ecd6e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.570760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.570741    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5" pod "20563428-9ca2-4c60-ab11-fb4930f95a61" (UID: "20563428-9ca2-4c60-ab11-fb4930f95a61"). InnerVolumeSpecName "kube-api-access-nl6p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.667854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667814    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.667854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667856    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667871    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667886    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667900    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20563428-9ca2-4c60-ab11-fb4930f95a61-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667913    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667927    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667941    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667954    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl6p5\" (UniqueName: \"kubernetes.io/projected/20563428-9ca2-4c60-ab11-fb4930f95a61-kube-api-access-nl6p5\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667970    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20563428-9ca2-4c60-ab11-fb4930f95a61-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667984    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.668103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.667997    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzn9j\" (UniqueName: \"kubernetes.io/projected/2793053a-78e8-43d1-87e3-9f753ecd6e89-kube-api-access-tzn9j\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.674351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.674312    5510 generic.go:356] "Generic (PLEG): container finished" podID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerID="ee2b2039225677ab6b38574840773e9b24445a0a985a0aa283011e49bb9d020c" exitCode=0
Oct 30 22:08:02.674531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.674388    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" event={"ID":"28488c09-4bf6-41ed-adbe-d19cb5cca34e","Type":"ContainerDied","Data":"ee2b2039225677ab6b38574840773e9b24445a0a985a0aa283011e49bb9d020c"}
Oct 30 22:08:02.676229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.676205    5510 generic.go:356] "Generic (PLEG): container finished" podID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerID="d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de" exitCode=0
Oct 30 22:08:02.676351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.676266    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"
Oct 30 22:08:02.676351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.676294    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" event={"ID":"20563428-9ca2-4c60-ab11-fb4930f95a61","Type":"ContainerDied","Data":"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"}
Oct 30 22:08:02.676469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.676350    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg" event={"ID":"20563428-9ca2-4c60-ab11-fb4930f95a61","Type":"ContainerDied","Data":"400e860450585927a37ac998cb60d5ae988d8f80588c2af049b2cb54855cd157"}
Oct 30 22:08:02.676469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.676383    5510 scope.go:122] "RemoveContainer" containerID="d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"
Oct 30 22:08:02.678283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.678260    5510 generic.go:356] "Generic (PLEG): container finished" podID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerID="83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e" exitCode=0
Oct 30 22:08:02.678378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.678313    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"
Oct 30 22:08:02.678378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.678370    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" event={"ID":"2793053a-78e8-43d1-87e3-9f753ecd6e89","Type":"ContainerDied","Data":"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"}
Oct 30 22:08:02.678507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.678444    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k" event={"ID":"2793053a-78e8-43d1-87e3-9f753ecd6e89","Type":"ContainerDied","Data":"c26f04cd53e4781a2e0f64adb8c0b83f955edcde08b43b17283575a7b91629b2"}
Oct 30 22:08:02.688585 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.688561    5510 scope.go:122] "RemoveContainer" containerID="d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"
Oct 30 22:08:02.688971 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:08:02.688940    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de\": container with ID starting with d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de not found: ID does not exist" containerID="d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"
Oct 30 22:08:02.689226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.688983    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de"} err="failed to get container status \"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de\": rpc error: code = NotFound desc = could not find container \"d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de\": container with ID starting with d848e677f64df67b592c24538e42e155eb4c7a8fa31ffd64af2a5b3608fb88de not found: ID does not exist"
Oct 30 22:08:02.689226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.689010    5510 scope.go:122] "RemoveContainer" containerID="83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"
Oct 30 22:08:02.706324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.706297    5510 scope.go:122] "RemoveContainer" containerID="83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"
Oct 30 22:08:02.706719 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:08:02.706689    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e\": container with ID starting with 83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e not found: ID does not exist" containerID="83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"
Oct 30 22:08:02.706839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.706731    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e"} err="failed to get container status \"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e\": rpc error: code = NotFound desc = could not find container \"83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e\": container with ID starting with 83f17b1ddc217b73090e7b5b918b5f4b34d182578c657d9c619e6072fe824f9e not found: ID does not exist"
Oct 30 22:08:02.850359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.850257    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:08:02.854808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.854769    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qnkcg"]
Oct 30 22:08:02.956395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.956351    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 22:08:02.962599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:02.962572    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-9th7k"]
Oct 30 22:08:03.140870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.135336    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" path="/var/lib/kubelet/pods/20563428-9ca2-4c60-ab11-fb4930f95a61/volumes"
Oct 30 22:08:03.140870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.136142    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" path="/var/lib/kubelet/pods/2793053a-78e8-43d1-87e3-9f753ecd6e89/volumes"
Oct 30 22:08:03.199481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.199455    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:08:03.277116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277084    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277156    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277182    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277211    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277267    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277325    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277408    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277498    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277527    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\") pod \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\" (UID: \"28488c09-4bf6-41ed-adbe-d19cb5cca34e\") "
Oct 30 22:08:03.277757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277582    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:03.277757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277620    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:03.277757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277741    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:03.278062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.277807    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:03.278062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278014    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:03.278461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278437    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.278543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278466    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.278543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278483    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.278543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278499    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.278543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.278515    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.279443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.279418    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:03.279546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.279506    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "kube-api-access-w6zkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:03.279645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.279625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:03.279822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.279801    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy" pod "28488c09-4bf6-41ed-adbe-d19cb5cca34e" (UID: "28488c09-4bf6-41ed-adbe-d19cb5cca34e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:03.379683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.379636    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zkq\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-kube-api-access-w6zkq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.379683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.379675    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.379683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.379685    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.379683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.379694    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28488c09-4bf6-41ed-adbe-d19cb5cca34e-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.688687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.688645    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"
Oct 30 22:08:03.688687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.688670    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc" event={"ID":"28488c09-4bf6-41ed-adbe-d19cb5cca34e","Type":"ContainerDied","Data":"68ceb007cdee21aaa7a136af6fe8324a3a3ec8fac73c7e7be1b045d1479454b5"}
Oct 30 22:08:03.688969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.688735    5510 scope.go:122] "RemoveContainer" containerID="ee2b2039225677ab6b38574840773e9b24445a0a985a0aa283011e49bb9d020c"
Oct 30 22:08:03.729218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.729185    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:03.880109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.880062    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:08:03.904942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:03.904900    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-qc9sc"]
Oct 30 22:08:05.127006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:05.126968    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" path="/var/lib/kubelet/pods/28488c09-4bf6-41ed-adbe-d19cb5cca34e/volumes"
Oct 30 22:08:08.288128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:08.288098    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:11.625077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:11.625049    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:08:11.635772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:11.635749    5510 kubelet.go:1689] "Image garbage collection succeeded"
Oct 30 22:08:11.930470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:11.930392    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:08:12.051191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:12.051156    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"]
Oct 30 22:08:12.097326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:12.097293    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9db5747-qfmg7"]
Oct 30 22:08:12.734173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:12.734142    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:13.128156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:13.128067    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f298f672-77a7-42ad-9531-5066a2229c6b" path="/var/lib/kubelet/pods/f298f672-77a7-42ad-9531-5066a2229c6b/volumes"
Oct 30 22:08:13.280836 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:08:13.280762    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:08:17.143272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.143172    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.323809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.317240    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.498806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.498761    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.553980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.553937    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"]
Oct 30 22:08:17.555012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.554987    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy"
Oct 30 22:08:17.555012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555012    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555030    5510 cpu_manager.go:414] "containerMap: removing container" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555039    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555064    5510 cpu_manager.go:414] "containerMap: removing container" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555072    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555097    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555104    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555126    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555133    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555149    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555156    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555168    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555175    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555343    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="20563428-9ca2-4c60-ab11-fb4930f95a61" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555357    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd66532-5265-43f5-a3d6-8a5db541364b" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555368    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555380    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b515f9-9ff1-4884-b543-4c138abc4258" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555389    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2793053a-78e8-43d1-87e3-9f753ecd6e89" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555402    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e31c469-5cab-4c32-83ba-aa5d70741061" containerName="istio-proxy"
Oct 30 22:08:17.555550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.555411    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="28488c09-4bf6-41ed-adbe-d19cb5cca34e" containerName="istio-proxy"
Oct 30 22:08:17.561421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.561398    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.603413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.603378    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"]
Oct 30 22:08:17.642899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.642864    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:17.643057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.642954    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:17.643127 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.643072    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:08:17.657584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.657535    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:17.750395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.750318    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.750541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.750479    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ghv\" (UniqueName: \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.750657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.750632    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.851579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.851543    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.851737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.851705    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ghv\" (UniqueName: \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.851875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.851858    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.852503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.852484    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.853987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.853956    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:17.905686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:17.905659    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ghv\" (UniqueName: \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\") pod \"maas-api-revoke-sub-76051f8a-vcszq\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:18.061437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:18.061348    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"]
Oct 30 22:08:18.065734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:18.065706    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c848b8c3-5zcrl"]
Oct 30 22:08:18.176013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:18.175976    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:18.555002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:18.554964    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"]
Oct 30 22:08:18.560644 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:08:18.560617    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b4f0ff_d5e9_4048_b49f_659a8233154b.slice/crio-d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca WatchSource:0}: Error finding container d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca: Status 404 returned error can't find the container with id d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca
Oct 30 22:08:18.892170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:18.892098    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq" event={"ID":"21b4f0ff-d5e9-4048-b49f-659a8233154b","Type":"ContainerStarted","Data":"d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca"}
Oct 30 22:08:19.139991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:19.139959    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7564101-1af7-44e7-80ab-acfe60e29f84" path="/var/lib/kubelet/pods/d7564101-1af7-44e7-80ab-acfe60e29f84/volumes"
Oct 30 22:08:19.915863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:19.915820    5510 generic.go:356] "Generic (PLEG): container finished" podID="21b4f0ff-d5e9-4048-b49f-659a8233154b" containerID="96ea9676c377918a2560fa2207019fc893ddf2e0e9465d52fcf3fa58c57772ec" exitCode=0
Oct 30 22:08:19.916285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:19.915899    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq" event={"ID":"21b4f0ff-d5e9-4048-b49f-659a8233154b","Type":"ContainerDied","Data":"96ea9676c377918a2560fa2207019fc893ddf2e0e9465d52fcf3fa58c57772ec"}
Oct 30 22:08:20.151876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:20.151834    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"]
Oct 30 22:08:20.158150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:20.158115    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a7968a8-bm9t4"]
Oct 30 22:08:21.122705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.122681    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:21.126385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.126359    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e705defa-1b51-4419-87d7-f954605fba90" path="/var/lib/kubelet/pods/e705defa-1b51-4419-87d7-f954605fba90/volumes"
Oct 30 22:08:21.169378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.169352    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\") pod \"21b4f0ff-d5e9-4048-b49f-659a8233154b\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") "
Oct 30 22:08:21.169573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.169560    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\" (UniqueName: \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\") pod \"21b4f0ff-d5e9-4048-b49f-659a8233154b\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") "
Oct 30 22:08:21.169630 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.169600    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\") pod \"21b4f0ff-d5e9-4048-b49f-659a8233154b\" (UID: \"21b4f0ff-d5e9-4048-b49f-659a8233154b\") "
Oct 30 22:08:21.169764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.169739    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca" pod "21b4f0ff-d5e9-4048-b49f-659a8233154b" (UID: "21b4f0ff-d5e9-4048-b49f-659a8233154b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:21.170318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.170285    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:21.171836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.171809    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv" pod "21b4f0ff-d5e9-4048-b49f-659a8233154b" (UID: "21b4f0ff-d5e9-4048-b49f-659a8233154b"). InnerVolumeSpecName "kube-api-access-57ghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:21.172105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.172083    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca" pod "21b4f0ff-d5e9-4048-b49f-659a8233154b" (UID: "21b4f0ff-d5e9-4048-b49f-659a8233154b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:21.272447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.272407    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ghv\" (UniqueName: \"kubernetes.io/projected/21b4f0ff-d5e9-4048-b49f-659a8233154b-kube-api-access-57ghv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:21.272447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.272449    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b4f0ff-d5e9-4048-b49f-659a8233154b-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:21.932746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.932715    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"
Oct 30 22:08:21.932910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.932714    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq" event={"ID":"21b4f0ff-d5e9-4048-b49f-659a8233154b","Type":"ContainerDied","Data":"d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca"}
Oct 30 22:08:21.932910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.932853    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d514c4a0613a180c1450a897c3fff4e54a1904d96419f667cb86037a07d0cbca"
Oct 30 22:08:21.961303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.961266    5510 scope.go:122] "RemoveContainer" containerID="264127c2700132114d63159236e863de6667a4fd046ae61ae58d383553c848b0"
Oct 30 22:08:21.971619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.971598    5510 scope.go:122] "RemoveContainer" containerID="8c327f5688b33e793423a45d2fb68f37f1cc4cc6fa8ab1e0da9d291ee2dddca7"
Oct 30 22:08:21.984874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:21.984849    5510 scope.go:122] "RemoveContainer" containerID="e69ab3569559166c48fba6b9efa6e0546ae17ba01927a97fc5d0b4f0a412d908"
Oct 30 22:08:22.092657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:22.082467    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:24.224512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:24.224481    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:25.062704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:25.062670    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"]
Oct 30 22:08:25.073304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:25.073275    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea873ee5-rstwf"]
Oct 30 22:08:25.126230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:25.126200    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68467546-4aa5-4445-82df-375d470f7d04" path="/var/lib/kubelet/pods/68467546-4aa5-4445-82df-375d470f7d04/volumes"
Oct 30 22:08:25.507884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:25.507851    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:26.049504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:26.049473    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"]
Oct 30 22:08:26.095399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:26.095370    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c8448e-pj4xw"]
Oct 30 22:08:26.857809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:26.857749    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:27.128246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:27.128175    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7331d156-e0f1-4d91-9363-7d5e31f8629c" path="/var/lib/kubelet/pods/7331d156-e0f1-4d91-9363-7d5e31f8629c/volumes"
Oct 30 22:08:28.326082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:28.326051    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:28.742979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:28.742947    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:29.062251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.062180    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"]
Oct 30 22:08:29.107970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.107941    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc2e78cb-sjb8x"]
Oct 30 22:08:29.214532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.214494    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f91442f-9ecf-4152-b46c-e90c2093a800" path="/var/lib/kubelet/pods/8f91442f-9ecf-4152-b46c-e90c2093a800/volumes"
Oct 30 22:08:29.314544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.314456    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:08:29.315699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.315671    5510 cpu_manager.go:414] "containerMap: removing container" podUID="21b4f0ff-d5e9-4048-b49f-659a8233154b" containerName="revoke-keys"
Oct 30 22:08:29.315915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.315893    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b4f0ff-d5e9-4048-b49f-659a8233154b" containerName="revoke-keys"
Oct 30 22:08:29.316268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.316251    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b4f0ff-d5e9-4048-b49f-659a8233154b" containerName="revoke-keys"
Oct 30 22:08:29.383738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.383686    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.444319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.444287    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448496    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448557    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448641    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q797b\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448676    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448719    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448852    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.448929    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.449008    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.449391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.449145    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.452982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.452217    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-4d8041-openshift-default-dockercfg-jlrmp\""
Oct 30 22:08:29.633922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.565183    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.689544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.689506    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.717589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.717563    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.731883    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.732072    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q797b\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.732115    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.732164    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.732342    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.732430    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.742266    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.570363    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.743830    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.680444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.690194    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.744998    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.757797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.753246    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.760938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.760904    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.793913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.793237    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.817246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.817213    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.819761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.819730    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q797b\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\") pod \"e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") " pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:29.989750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:29.989717    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:30.431656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:30.431520    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:08:30.437066 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:08:30.437035    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544ea754_1458_4660_939e_4f974fc3cd17.slice/crio-9db94a473a818a5fce4599fef338d3748eebdac5e9372b0e7fd1bf43a45fecff WatchSource:0}: Error finding container 9db94a473a818a5fce4599fef338d3748eebdac5e9372b0e7fd1bf43a45fecff: Status 404 returned error can't find the container with id 9db94a473a818a5fce4599fef338d3748eebdac5e9372b0e7fd1bf43a45fecff
Oct 30 22:08:30.439623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:30.439591    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:30.439722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:30.439663    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:30.439722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:30.439711    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:31.020820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.020752    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerStarted","Data":"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"}
Oct 30 22:08:31.021006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.020879    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerStarted","Data":"9db94a473a818a5fce4599fef338d3748eebdac5e9372b0e7fd1bf43a45fecff"}
Oct 30 22:08:31.209044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.208987    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podStartSLOduration=2.208968755 podStartE2EDuration="2.208968755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:31.179729481 +0000 UTC m=+1520.163636356" watchObservedRunningTime="2026-10-30 22:08:31.208968755 +0000 UTC m=+1520.192875635"
Oct 30 22:08:31.660043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.659410    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:31.990316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.990283    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:31.996206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:31.996175    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:32.028142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:32.028108    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:32.029057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:32.029037    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:08:34.759741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:34.759696    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:34.966526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:34.966490    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:35.208137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:35.208106    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:36.580978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:36.580949    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:36.747688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:36.746426    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:38.082125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.082083    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:38.348862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.339901    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:38.693477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.693376    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:08:38.712746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.712712    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:38.775802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.759015    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:08:38.779643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.779431    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-4d8041\""
Oct 30 22:08:38.786854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.780231    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-4d8041-dockercfg-vrr6r\""
Oct 30 22:08:38.867835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.839536    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:38.867835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.839866    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5cv\" (UniqueName: \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:38.882447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:38.882406    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:08:39.177854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.078828    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5cv\" (UniqueName: \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:39.205662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.205623    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:39.249840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.248957    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.275998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.275956    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:39.279463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.279437    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5cv\" (UniqueName: \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\") pod \"payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:39.306154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.306125    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:08:39.330051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.330017    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svg8\" (UniqueName: \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.330174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.330060    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.431938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.431838    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svg8\" (UniqueName: \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.432118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.431956    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.432746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.432722    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.490877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.490848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svg8\" (UniqueName: \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\") pod \"payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.517041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.517013    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:39.617377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.617340    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:39.902628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:39.902603    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:08:39.905314 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:08:39.905281    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e40124_3293_439d_89d0_082236c776d1.slice/crio-6af371323b3613ae5d120a4d549098d699eebe6f981f1e01ceb95ce41a710b44 WatchSource:0}: Error finding container 6af371323b3613ae5d120a4d549098d699eebe6f981f1e01ceb95ce41a710b44: Status 404 returned error can't find the container with id 6af371323b3613ae5d120a4d549098d699eebe6f981f1e01ceb95ce41a710b44
Oct 30 22:08:40.022059 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:08:40.022024    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfb9742_e8af_4b16_91c8_01e9b08c84b3.slice/crio-4c12e33c6a21875e39d72bd5ab9e04b84f600ec3d5abfcdf4ae7aa1597a35fb0 WatchSource:0}: Error finding container 4c12e33c6a21875e39d72bd5ab9e04b84f600ec3d5abfcdf4ae7aa1597a35fb0: Status 404 returned error can't find the container with id 4c12e33c6a21875e39d72bd5ab9e04b84f600ec3d5abfcdf4ae7aa1597a35fb0
Oct 30 22:08:40.022200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.022117    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:08:40.285582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.285543    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" event={"ID":"d5e40124-3293-439d-89d0-082236c776d1","Type":"ContainerStarted","Data":"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"}
Oct 30 22:08:40.286125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.285612    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" event={"ID":"d5e40124-3293-439d-89d0-082236c776d1","Type":"ContainerStarted","Data":"6af371323b3613ae5d120a4d549098d699eebe6f981f1e01ceb95ce41a710b44"}
Oct 30 22:08:40.286125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.285645    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:40.287856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.287822    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" event={"ID":"3cfb9742-e8af-4b16-91c8-01e9b08c84b3","Type":"ContainerStarted","Data":"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"}
Oct 30 22:08:40.287992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.287886    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:40.287992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.287911    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" event={"ID":"3cfb9742-e8af-4b16-91c8-01e9b08c84b3","Type":"ContainerStarted","Data":"4c12e33c6a21875e39d72bd5ab9e04b84f600ec3d5abfcdf4ae7aa1597a35fb0"}
Oct 30 22:08:40.395721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.395662    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" podStartSLOduration=2.395648229 podStartE2EDuration="2.395648229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:40.393645597 +0000 UTC m=+1529.377552473" watchObservedRunningTime="2026-10-30 22:08:40.395648229 +0000 UTC m=+1529.379555105"
Oct 30 22:08:40.544722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:40.544597    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" podStartSLOduration=2.544576314 podStartE2EDuration="2.544576314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:40.519244277 +0000 UTC m=+1529.503151153" watchObservedRunningTime="2026-10-30 22:08:40.544576314 +0000 UTC m=+1529.528483192"
Oct 30 22:08:41.551976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:41.551938    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:43.281195 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:08:43.281152    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:08:43.830802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:43.822228    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:43.965830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:43.963370    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:08:46.298159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:46.298120    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:47.077804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:47.076892    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:08:47.157806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:47.157457    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:47.179804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:47.177706    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:47.215801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:47.214920    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:47.776806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:47.772953    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.023310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.022540    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.079015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.077055    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.079015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.077424    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:08:48.087184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.086573    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:08:48.087184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.086876    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.166807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.157855    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.166807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.158471    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:08:48.166807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.159909    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.166807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.160712    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.184813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.184571    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.185020 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.185005    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:08:48.226805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.223601    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.226805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.225016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:08:48.306863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.304518    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:08:48.395858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.345215    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:48.635119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:48.635071    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:49.127806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.114726    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.127806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.115759    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.138410    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.138678    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.138737    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.162744    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.163030    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.195864    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.214355    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:49.267806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.233644    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.445725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.398855    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:08:49.451607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.451567    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:49.468632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.468602    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:08:49.468881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.468866    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:08:49.778806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:49.757433    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.103324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.103215    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.149970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.149900    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:08:50.288897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.288035    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:08:50.308819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.290851    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:08:50.308819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.291544    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.308819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.292315    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.360860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.330738    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:08:50.363700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.351475    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:08:50.439927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.439868    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.465807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.440729    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:08:50.526603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.526203    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-x8gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:50.526954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.526898    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.712748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.712706    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.771317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.771274    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:50.773072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.773046    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:08:50.934768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:50.934734    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:51.215810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.214310    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.295891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.289806    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.300317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.300278    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.316056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.316017    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.334796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.319536    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.382009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.381945    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:51.405153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.405105    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:51.486106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.486010    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 30 22:08:51.487651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.487599    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:51.555588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.555550    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:08:51.555898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.555880    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:08:51.556080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.556067    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:08:51.556220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.556208    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:08:51.802861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:51.785701    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:08:52.233467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.228613    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:08:52.291329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.288871    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:08:52.360804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.357195    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:52.384800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.384748    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 22:08:52.441277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.432499    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:08:52.441277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.421387    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:52.441277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.434303    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:08:52.441277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.434893    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:52.442925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.442670    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:52.530221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.530143    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:08:52.543511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.543482    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:08:52.546992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.546950    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:08:52.611104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.611071    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:08:52.611372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.611353    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"]
Oct 30 22:08:52.629518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.629489    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:08:52.637293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:52.637251    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc4d9-x9b5l"]
Oct 30 22:08:53.085128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:53.085092    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"]
Oct 30 22:08:53.193003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:53.192949    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9beb9e4a-e922-4ae2-bccc-77039a16dd44" path="/var/lib/kubelet/pods/9beb9e4a-e922-4ae2-bccc-77039a16dd44/volumes"
Oct 30 22:08:53.193700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:53.193677    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603a71df-kftlh"]
Oct 30 22:08:54.629154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:54.629034    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:55.115727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:55.115653    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:55.171023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:55.170987    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68c5da5-3b11-4a61-899f-c518d2a3820f" path="/var/lib/kubelet/pods/b68c5da5-3b11-4a61-899f-c518d2a3820f/volumes"
Oct 30 22:08:55.721074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:55.721043    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:56.361363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:56.361321    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:59.720276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.720244    5510 generic.go:356] "Generic (PLEG): container finished" podID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerID="510132f7fd014b77c2c92fba747d84ca404fa3be52d4111a1baa89748274c963" exitCode=0
Oct 30 22:08:59.720843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.720320    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerDied","Data":"510132f7fd014b77c2c92fba747d84ca404fa3be52d4111a1baa89748274c963"}
Oct 30 22:08:59.720843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.720776    5510 scope.go:122] "RemoveContainer" containerID="510132f7fd014b77c2c92fba747d84ca404fa3be52d4111a1baa89748274c963"
Oct 30 22:08:59.722307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722287    5510 generic.go:356] "Generic (PLEG): container finished" podID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerID="9356071a480c1740cfe04edc3bb11bf39f162c9b67c4a9184a356df492163a23" exitCode=0
Oct 30 22:08:59.722435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722334    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerDied","Data":"9356071a480c1740cfe04edc3bb11bf39f162c9b67c4a9184a356df492163a23"}
Oct 30 22:08:59.722515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722455    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.722589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722518    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.722589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722551    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.722738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.722721    5510 scope.go:122] "RemoveContainer" containerID="9356071a480c1740cfe04edc3bb11bf39f162c9b67c4a9184a356df492163a23"
Oct 30 22:08:59.724276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.724250    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.724361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.724318    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.724361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.724350    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:08:59.983304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:08:59.983214    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:00.732444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:00.732168    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerStarted","Data":"0c0b7cd16205864a440db233ca326f2b38d7084df26b4d7f8da741f09fdbdb24"}
Oct 30 22:09:00.734379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:00.734355    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerStarted","Data":"ab7f064324e56cc670bc8e43a1fa02e046bdf0c0daef04e2e9a7c0214f43cdc0"}
Oct 30 22:09:00.795461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:00.795427    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:09:00.803884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:00.801977    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:00.803884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:00.802040    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:01.346557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.346522    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:01.578806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.578194    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:01.590804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.583890    5510 patch_prober.go:29] interesting pod/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:09:01.590804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.583959    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:09:01.790803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.790148    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:01.790803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:01.790210    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:02.108239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.107695    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"]
Oct 30 22:09:02.115764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.115729    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4757509e-9nvhj"]
Oct 30 22:09:02.593801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.588259    5510 patch_prober.go:29] interesting pod/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:09:02.593801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.588330    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:09:02.811806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.806994    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:02.811806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.807054    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:02.939757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:02.939680    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:03.131863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.131774    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3" path="/var/lib/kubelet/pods/2ebc0c7b-9ed5-4437-a8ca-c9f575848fc3/volumes"
Oct 30 22:09:03.592804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.589323    5510 patch_prober.go:29] interesting pod/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:09:03.592804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.589405    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:09:03.673807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.663236    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"]
Oct 30 22:09:03.681817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.680095    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.714016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.713982    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"]
Oct 30 22:09:03.747165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.747138    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:03.747347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.747177    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:03.793804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.779945    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:03.793804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.780064    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:09:03.800933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.800897    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:03.801119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.800957    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:03.814886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.814854    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.815294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.814902    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f9x\" (UniqueName: \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.815294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.815051    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.916713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.916631    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.916904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.916851    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.916904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.916888    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f9x\" (UniqueName: \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.917383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.917359    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.919429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.919406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:03.998805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:03.986214    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f9x\" (UniqueName: \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\") pod \"maas-api-revoke-keys-e2e-migrate-13297892-g4wpn\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:04.016370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.016333    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:04.506357 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:09:04.506301    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe5ab3b_f9fd_4113_818d_9d1da2895554.slice/crio-f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d WatchSource:0}: Error finding container f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d: Status 404 returned error can't find the container with id f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d
Oct 30 22:09:04.509095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.509017    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"]
Oct 30 22:09:04.580815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.578214    5510 patch_prober.go:29] interesting pod/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:09:04.580815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.578278    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:09:04.790619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.790162    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:04.790619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.790260    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:04.822119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:04.822073    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn" event={"ID":"bbe5ab3b-f9fd-4113-818d-9d1da2895554","Type":"ContainerStarted","Data":"f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d"}
Oct 30 22:09:05.580215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.580173    5510 patch_prober.go:29] interesting pod/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:09:05.580545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.580513    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 22:09:05.646041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.646003    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:09:05.791923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.791883    5510 patch_prober.go:29] interesting pod/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:09:05.792219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.792185    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 22:09:05.835933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.834921    5510 generic.go:356] "Generic (PLEG): container finished" podID="bbe5ab3b-f9fd-4113-818d-9d1da2895554" containerID="cb7447d3ca757c29eb6dc4078d7c789ca9c7eb34bc0cf889b9742c2a943154af" exitCode=0
Oct 30 22:09:05.835933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:05.835010    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn" event={"ID":"bbe5ab3b-f9fd-4113-818d-9d1da2895554","Type":"ContainerDied","Data":"cb7447d3ca757c29eb6dc4078d7c789ca9c7eb34bc0cf889b9742c2a943154af"}
Oct 30 22:09:06.115870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.114347    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:06.583075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.583041    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:06.583533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.583511    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:06.584187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.584168    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:06.794635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.794604    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:09:06.795853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:06.795827    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:09:07.114761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.114591    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:07.219208 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.219177    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\" (UniqueName: \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\") pod \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") "
Oct 30 22:09:07.219392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.219271    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\") pod \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") "
Oct 30 22:09:07.219457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.219444    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\") pod \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\" (UID: \"bbe5ab3b-f9fd-4113-818d-9d1da2895554\") "
Oct 30 22:09:07.219630 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.219603    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca" pod "bbe5ab3b-f9fd-4113-818d-9d1da2895554" (UID: "bbe5ab3b-f9fd-4113-818d-9d1da2895554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:07.220052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.220031    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:07.221994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.221936    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca" pod "bbe5ab3b-f9fd-4113-818d-9d1da2895554" (UID: "bbe5ab3b-f9fd-4113-818d-9d1da2895554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:07.222437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.222411    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x" pod "bbe5ab3b-f9fd-4113-818d-9d1da2895554" (UID: "bbe5ab3b-f9fd-4113-818d-9d1da2895554"). InnerVolumeSpecName "kube-api-access-v7f9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:07.321939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.321902    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7f9x\" (UniqueName: \"kubernetes.io/projected/bbe5ab3b-f9fd-4113-818d-9d1da2895554-kube-api-access-v7f9x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:07.321939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.321935    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe5ab3b-f9fd-4113-818d-9d1da2895554-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:07.852841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.852813    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"
Oct 30 22:09:07.853018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.852833    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn" event={"ID":"bbe5ab3b-f9fd-4113-818d-9d1da2895554","Type":"ContainerDied","Data":"f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d"}
Oct 30 22:09:07.853018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:07.852894    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8930071fdd05bf4517df37c738c0ae0ecd4ea5e339519f9efab09982df5fd7d"
Oct 30 22:09:09.334871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:09.334837    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:10.053317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:10.053274    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"]
Oct 30 22:09:10.100186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:10.100151    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca202331-5gg4s"]
Oct 30 22:09:10.935813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:10.930656    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:11.077116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:11.077079    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"]
Oct 30 22:09:11.129906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:11.129879    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66874041-5ef0-4dbe-8ff2-fdace8f3f5ba" path="/var/lib/kubelet/pods/66874041-5ef0-4dbe-8ff2-fdace8f3f5ba/volumes"
Oct 30 22:09:11.130221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:11.130197    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"]
Oct 30 22:09:11.170213 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:09:11.170168    5510 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-212c951d-r852w\" is forbidden: User \"system:node:ip-10-0-131-231.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-231.us-west-2.compute.internal' and this object" podUID="f1e49b28-2047-4351-9f85-60a31a5d91ba" pod="odh-ai-gateway-infra/maas-api-revoke-sub-212c951d-r852w"
Oct 30 22:09:12.953925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:12.953890    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"]
Oct 30 22:09:12.954906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:12.954878    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bbe5ab3b-f9fd-4113-818d-9d1da2895554" containerName="revoke-keys"
Oct 30 22:09:12.954906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:12.954902    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe5ab3b-f9fd-4113-818d-9d1da2895554" containerName="revoke-keys"
Oct 30 22:09:12.955107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:12.955095    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe5ab3b-f9fd-4113-818d-9d1da2895554" containerName="revoke-keys"
Oct 30 22:09:12.960030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:12.960011    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.012349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.012316    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"]
Oct 30 22:09:13.023054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.023020    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.023221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.023192    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.023385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.023367    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcc6\" (UniqueName: \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.053088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.053039    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:13.053252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.053095    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:09:13.056110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.056087    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:13.060709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.056544    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:13.124386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.124356    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.124567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.124551    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcc6\" (UniqueName: \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.124698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.124665    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.125195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.125160    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.127167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.127141    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.127624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.127597    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e49b28-2047-4351-9f85-60a31a5d91ba" path="/var/lib/kubelet/pods/f1e49b28-2047-4351-9f85-60a31a5d91ba/volumes"
Oct 30 22:09:13.178510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.178473    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcc6\" (UniqueName: \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\") pod \"maas-api-revoke-keys-e2e-derive-de9c059f-rkztv\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.274910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.274875    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:13.284965 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:09:13.284932    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:09:13.661827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.661668    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"]
Oct 30 22:09:13.667904 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:09:13.667874    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161152ea_a97b_4a35_885f_fed6358839f3.slice/crio-95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c WatchSource:0}: Error finding container 95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c: Status 404 returned error can't find the container with id 95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c
Oct 30 22:09:13.912831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:13.912746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv" event={"ID":"161152ea-a97b-4a35-885f-fed6358839f3","Type":"ContainerStarted","Data":"95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c"}
Oct 30 22:09:14.922090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:14.922007    5510 generic.go:356] "Generic (PLEG): container finished" podID="161152ea-a97b-4a35-885f-fed6358839f3" containerID="21ab180ccffafb67d81fd76ffc9805c9e52e2b620521dbb6dd8ee7a101acfbc8" exitCode=0
Oct 30 22:09:14.922461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:14.922095    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv" event={"ID":"161152ea-a97b-4a35-885f-fed6358839f3","Type":"ContainerDied","Data":"21ab180ccffafb67d81fd76ffc9805c9e52e2b620521dbb6dd8ee7a101acfbc8"}
Oct 30 22:09:15.607617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:15.607581    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:15.787741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:15.787690    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:15.985620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:15.985565    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.153494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.153461    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.325359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.325334    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:16.433679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.433648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\") pod \"161152ea-a97b-4a35-885f-fed6358839f3\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") "
Oct 30 22:09:16.433872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.433769    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\") pod \"161152ea-a97b-4a35-885f-fed6358839f3\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") "
Oct 30 22:09:16.434132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.434106    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\" (UniqueName: \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\") pod \"161152ea-a97b-4a35-885f-fed6358839f3\" (UID: \"161152ea-a97b-4a35-885f-fed6358839f3\") "
Oct 30 22:09:16.434274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.434139    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca" pod "161152ea-a97b-4a35-885f-fed6358839f3" (UID: "161152ea-a97b-4a35-885f-fed6358839f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.434944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.434919    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.438905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.438877    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca" pod "161152ea-a97b-4a35-885f-fed6358839f3" (UID: "161152ea-a97b-4a35-885f-fed6358839f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:16.439072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.438914    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6" pod "161152ea-a97b-4a35-885f-fed6358839f3" (UID: "161152ea-a97b-4a35-885f-fed6358839f3"). InnerVolumeSpecName "kube-api-access-mzcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.536068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.536035    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161152ea-a97b-4a35-885f-fed6358839f3-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.536068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.536064    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcc6\" (UniqueName: \"kubernetes.io/projected/161152ea-a97b-4a35-885f-fed6358839f3-kube-api-access-mzcc6\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.810634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:16.810562    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:17.099854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:17.099190    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv" event={"ID":"161152ea-a97b-4a35-885f-fed6358839f3","Type":"ContainerDied","Data":"95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c"}
Oct 30 22:09:17.099854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:17.099255    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95bec339b0cb33ad474cc7dcd4daa41ec39f9ecddf9482deb05b76f2a437465c"
Oct 30 22:09:17.099854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:17.099341    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"
Oct 30 22:09:19.154823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:19.154787    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:19.317088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:19.317054    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:19.482748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:19.482714    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:20.621566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:20.621519    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:22.178009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.177978    5510 scope.go:122] "RemoveContainer" containerID="a26f611bca362387f4ac2004c9d96c522a9264f006858407fdfbc3cf3fca6e17"
Oct 30 22:09:22.191025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.190999    5510 scope.go:122] "RemoveContainer" containerID="fb4f06a8aac75b0cd04879ba017fce3f6b797cdfa00a5ce63b44571844617f2e"
Oct 30 22:09:22.203075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.203049    5510 scope.go:122] "RemoveContainer" containerID="ccd44990e52eedb1909cb52d44b5ef49e5c25186b0c934ffe05d1082409a3f74"
Oct 30 22:09:22.213493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.213471    5510 scope.go:122] "RemoveContainer" containerID="88fbbb51e55bbad1bdb5b1122fcb11843634f66cdc3ddb2bd582d4b2cd324545"
Oct 30 22:09:22.218761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.218734    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:22.222869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.222849    5510 scope.go:122] "RemoveContainer" containerID="bb319eb8fcb4f1e662eded05c42c7b391d4f145c7d8faab671266d93932f57c0"
Oct 30 22:09:22.523819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.516662    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:22.831860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.828616    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:09:22.969804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.969173    5510 cpu_manager.go:414] "containerMap: removing container" podUID="161152ea-a97b-4a35-885f-fed6358839f3" containerName="revoke-keys"
Oct 30 22:09:22.969804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.969206    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161152ea-a97b-4a35-885f-fed6358839f3" containerName="revoke-keys"
Oct 30 22:09:22.969804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:22.969351    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="161152ea-a97b-4a35-885f-fed6358839f3" containerName="revoke-keys"
Oct 30 22:09:23.058507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.048459    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.064972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.064046    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:09:23.127801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.124027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.127801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.124365    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsxg\" (UniqueName: \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.158962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.158930    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7d434591\""
Oct 30 22:09:23.159136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.158990    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7d434591-dockercfg-ppccn\""
Oct 30 22:09:23.226387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.226356    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsxg\" (UniqueName: \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.226798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.226486    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.227207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.227188    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.280161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.280124    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsxg\" (UniqueName: \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\") pod \"payload-processing-e2e-mt-7d434591-5d5665c886-l6prx\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.441701 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.441609    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:09:23.457945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.457906    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:23.801646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.801610    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:23.954009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:23.952680    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:09:24.001802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:24.001595    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:24.105396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:24.102143    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"]
Oct 30 22:09:24.148723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:24.148686    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef19fa3-cxpjc"]
Oct 30 22:09:24.203799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:24.203710    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" event={"ID":"966ea468-60a5-4c66-8243-a008d56cca2a","Type":"ContainerStarted","Data":"edd73fc760df04214951812d96b039d6eef1d2b51f3931d50f3dea6649687223"}
Oct 30 22:09:25.260804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.259349    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7171a3bf-89f7-4b20-a837-f1caa7d08c23" path="/var/lib/kubelet/pods/7171a3bf-89f7-4b20-a837-f1caa7d08c23/volumes"
Oct 30 22:09:25.308024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.307982    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:25.315912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.315856    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" event={"ID":"966ea468-60a5-4c66-8243-a008d56cca2a","Type":"ContainerStarted","Data":"15caab9d76829ddd2ee7c36a9a2a23d7b19967e782cd2363db8bd770719cc7e5"}
Oct 30 22:09:25.331416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.331388    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:25.545830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.532630    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:25.561426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.561358    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" podStartSLOduration=3.561338086 podStartE2EDuration="3.561338086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:25.504291091 +0000 UTC m=+1574.488197969" watchObservedRunningTime="2026-10-30 22:09:25.561338086 +0000 UTC m=+1574.545244965"
Oct 30 22:09:25.757860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.753997    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:25.926290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:25.926195    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:27.048819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:27.048770    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"]
Oct 30 22:09:27.093989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:27.093949    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a4b2d0a-r9jkq"]
Oct 30 22:09:27.128324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:27.128293    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6013d903-973f-40ca-b72a-f6affaf888bc" path="/var/lib/kubelet/pods/6013d903-973f-40ca-b72a-f6affaf888bc/volumes"
Oct 30 22:09:28.109388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.109336    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:28.446690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.446589    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" podUID="dd7733fa-df76-4920-91c6-a92b58bfa294" containerName="payload-processing" containerID="cri-o://0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b" gracePeriod=25
Oct 30 22:09:28.738730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.738706    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:09:28.841126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.841092    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\") pod \"dd7733fa-df76-4920-91c6-a92b58bfa294\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") "
Oct 30 22:09:28.841294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.841204    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\" (UniqueName: \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\") pod \"dd7733fa-df76-4920-91c6-a92b58bfa294\" (UID: \"dd7733fa-df76-4920-91c6-a92b58bfa294\") "
Oct 30 22:09:28.841476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.841453    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume" pod "dd7733fa-df76-4920-91c6-a92b58bfa294" (UID: "dd7733fa-df76-4920-91c6-a92b58bfa294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:28.841878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.841863    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7733fa-df76-4920-91c6-a92b58bfa294-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:28.843421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.843393    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq" pod "dd7733fa-df76-4920-91c6-a92b58bfa294" (UID: "dd7733fa-df76-4920-91c6-a92b58bfa294"). InnerVolumeSpecName "kube-api-access-jf6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:28.943846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:28.943808    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6wq\" (UniqueName: \"kubernetes.io/projected/dd7733fa-df76-4920-91c6-a92b58bfa294-kube-api-access-jf6wq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:29.376743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.376707    5510 generic.go:356] "Generic (PLEG): container finished" podID="dd7733fa-df76-4920-91c6-a92b58bfa294" containerID="0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b" exitCode=0
Oct 30 22:09:29.377243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.376762    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"
Oct 30 22:09:29.377243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.376807    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" event={"ID":"dd7733fa-df76-4920-91c6-a92b58bfa294","Type":"ContainerDied","Data":"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"}
Oct 30 22:09:29.377243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.376862    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf" event={"ID":"dd7733fa-df76-4920-91c6-a92b58bfa294","Type":"ContainerDied","Data":"a81469f275d1ab382bfdf073a4757d263260e9767b3f7d802fcb08ab5ec89e01"}
Oct 30 22:09:29.377243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.376884    5510 scope.go:122] "RemoveContainer" containerID="0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"
Oct 30 22:09:29.387727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.387712    5510 scope.go:122] "RemoveContainer" containerID="0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"
Oct 30 22:09:29.388043 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:09:29.388022    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b\": container with ID starting with 0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b not found: ID does not exist" containerID="0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"
Oct 30 22:09:29.388125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.388052    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b"} err="failed to get container status \"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b\": rpc error: code = NotFound desc = could not find container \"0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b\": container with ID starting with 0d8f42d327535beb29066c402e8ed71776dd8d8c57c4a2255f646fea0ab9b00b not found: ID does not exist"
Oct 30 22:09:29.531834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.531802    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:09:29.577613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:29.577582    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-de9c059f-59677bf8-lhtbf"]
Oct 30 22:09:30.023086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:30.023025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.126513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:31.126475    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7733fa-df76-4920-91c6-a92b58bfa294" path="/var/lib/kubelet/pods/dd7733fa-df76-4920-91c6-a92b58bfa294/volumes"
Oct 30 22:09:34.538644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:34.538611    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:35.060431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:35.060386    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"]
Oct 30 22:09:35.103553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:35.103522    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee74d0b-v5htr"]
Oct 30 22:09:35.128801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:35.128755    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e46a775-ff97-4343-a067-b16e1a884f93" path="/var/lib/kubelet/pods/7e46a775-ff97-4343-a067-b16e1a884f93/volumes"
Oct 30 22:09:36.327347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:36.327319    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:09:37.042809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:37.042752    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"]
Oct 30 22:09:37.088755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:37.088721    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f197684-vdxnc"]
Oct 30 22:09:37.127243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:37.127203    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988aff3b-a219-4a4e-b210-c2aa6e4768fb" path="/var/lib/kubelet/pods/988aff3b-a219-4a4e-b210-c2aa6e4768fb/volumes"
Oct 30 22:09:38.411667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:38.411635    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:39.822649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:39.822600    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.040293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.040260    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.210261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.210210    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.288310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.288263    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"]
Oct 30 22:09:40.331237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.331198    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-13297892-g4wpn"]
Oct 30 22:09:40.352531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.352503    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:09:40.352946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.352917    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy" containerID="cri-o://ab7f064324e56cc670bc8e43a1fa02e046bdf0c0daef04e2e9a7c0214f43cdc0" gracePeriod=30
Oct 30 22:09:40.423663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.423624    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:40.897770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:40.889755    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:09:41.127352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.127315    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe5ab3b-f9fd-4113-818d-9d1da2895554" path="/var/lib/kubelet/pods/bbe5ab3b-f9fd-4113-818d-9d1da2895554/volumes"
Oct 30 22:09:41.262157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.262122    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:41.994423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.994389    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:09:41.996307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.996272    5510 cpu_manager.go:414] "containerMap: removing container" podUID="dd7733fa-df76-4920-91c6-a92b58bfa294" containerName="payload-processing"
Oct 30 22:09:41.996478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.996464    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7733fa-df76-4920-91c6-a92b58bfa294" containerName="payload-processing"
Oct 30 22:09:41.996926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:41.996908    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7733fa-df76-4920-91c6-a92b58bfa294" containerName="payload-processing"
Oct 30 22:09:42.027320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.027288    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.065469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.064276    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:42.070220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.068896    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:09:42.082498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.078668    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.083733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.083700    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089136    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089222    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089261    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089288    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwsf\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089328    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089365    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.090603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.089439    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.115806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.091859    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4215ed-openshift-default-dockercfg-jmv28\""
Oct 30 22:09:42.195958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.195925    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.196345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.196319    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.196517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.196499    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.196689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.196667    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.196909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.196889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.197054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.197036    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwsf\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.197215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.197194    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.197351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.197336    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.197495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.197480    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.199812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.199773    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.200458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.200434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.206436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.206409    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.206622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.206424    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.206905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.206877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.324039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.323923    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:42.343723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.340970    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.352845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.352766    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwsf\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.363726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.363691    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.364454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.364429    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\") pod \"e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") " pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.414069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.414024    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:42.888848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.888816    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:09:42.894486 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:09:42.894459    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42b7201_caa0_4e78_b1ef_c7f89630891b.slice/crio-25ae67c8f41ed1f5afe211e3d08f17a4edeff38a748945f476e11637f79b7a6c WatchSource:0}: Error finding container 25ae67c8f41ed1f5afe211e3d08f17a4edeff38a748945f476e11637f79b7a6c: Status 404 returned error can't find the container with id 25ae67c8f41ed1f5afe211e3d08f17a4edeff38a748945f476e11637f79b7a6c
Oct 30 22:09:42.896948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.896918    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:42.897072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.896985    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:42.897072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:42.897013    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:43.307944 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:09:43.307905    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:09:43.497707 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:43.497667    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerStarted","Data":"e85ce77043b8aff34969045c94cbc522c79b126471611190a18616ffe9fe6ed4"}
Oct 30 22:09:43.497897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:43.497735    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerStarted","Data":"25ae67c8f41ed1f5afe211e3d08f17a4edeff38a748945f476e11637f79b7a6c"}
Oct 30 22:09:43.711945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:43.711900    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podStartSLOduration=2.711886147 podStartE2EDuration="2.711886147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:43.663252109 +0000 UTC m=+1592.647158987" watchObservedRunningTime="2026-10-30 22:09:43.711886147 +0000 UTC m=+1592.695793022"
Oct 30 22:09:44.415949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:44.415911    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:44.422978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:44.422946    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:44.508390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:44.508361    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:44.509367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:44.509342    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:09:45.061992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.061892    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:45.519939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.519905    5510 generic.go:356] "Generic (PLEG): container finished" podID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerID="ab7f064324e56cc670bc8e43a1fa02e046bdf0c0daef04e2e9a7c0214f43cdc0" exitCode=0
Oct 30 22:09:45.520396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.519965    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerDied","Data":"ab7f064324e56cc670bc8e43a1fa02e046bdf0c0daef04e2e9a7c0214f43cdc0"}
Oct 30 22:09:45.520396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.520038    5510 scope.go:122] "RemoveContainer" containerID="9356071a480c1740cfe04edc3bb11bf39f162c9b67c4a9184a356df492163a23"
Oct 30 22:09:45.729429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.729393    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:45.788586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788530    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.788759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788675    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.788759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788748    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.788916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788836    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.788916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788906    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.789030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788930    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.789030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.788989    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.789030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789006    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.789193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789064    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.789193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789135    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.789303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789201    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.789303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789218    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\") pod \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\" (UID: \"8d2b9342-f6e2-4150-a585-e7db7b2a9a64\") "
Oct 30 22:09:45.789303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.789521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.789485    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:45.790050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.790029    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.790158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.790053    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.790158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.790068    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.790158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.790082    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.790158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.790096    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.791560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.791539    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:45.791837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.791775    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.792164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.792139    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:45.792266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.792222    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf" pod "8d2b9342-f6e2-4150-a585-e7db7b2a9a64" (UID: "8d2b9342-f6e2-4150-a585-e7db7b2a9a64"). InnerVolumeSpecName "kube-api-access-vvhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:45.897391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.894344    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.897391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.894383    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.897391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.894398    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhbf\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-kube-api-access-vvhbf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.897391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:45.894414    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2b9342-f6e2-4150-a585-e7db7b2a9a64-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:46.462804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.461189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:46.530870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.530834    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"
Oct 30 22:09:46.530870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.530834    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b" event={"ID":"8d2b9342-f6e2-4150-a585-e7db7b2a9a64","Type":"ContainerDied","Data":"d9317e1c6b26f4c948d5b90339d22f106d605fc810769eb857c43035af22adf4"}
Oct 30 22:09:46.531369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.530934    5510 scope.go:122] "RemoveContainer" containerID="ab7f064324e56cc670bc8e43a1fa02e046bdf0c0daef04e2e9a7c0214f43cdc0"
Oct 30 22:09:46.617485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.617452    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:46.740682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.740599    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:09:46.744383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.744353    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-13297892-gw-openshift-default-8cb4bbcc-qzq9b"]
Oct 30 22:09:46.782803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:46.782378    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:47.128142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:47.128075    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" path="/var/lib/kubelet/pods/8d2b9342-f6e2-4150-a585-e7db7b2a9a64/volumes"
Oct 30 22:09:49.581209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:49.581171    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:49.853612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:49.837411    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:50.031395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.031359    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:50.036684 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.036627    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:09:50.038636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.038607    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.039711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.039687    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.040248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.040231    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.041088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.041069    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.041567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.041533    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.085421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.084946    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.194853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.188908    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4215ed\""
Oct 30 22:09:50.194853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.194227    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4215ed-dockercfg-wgpwn\""
Oct 30 22:09:50.402240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.402198    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:09:50.477897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.444660    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:09:50.482512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.475842    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.482953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.482928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hw8\" (UniqueName: \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.560804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.552258    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2b9342-f6e2-4150-a585-e7db7b2a9a64" containerName="istio-proxy"
Oct 30 22:09:50.609805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.604879    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.609805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.605049    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hw8\" (UniqueName: \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.623810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.620725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.701997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.701959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hw8\" (UniqueName: \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\") pod \"payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.703122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.703099    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:09:50.703337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.703312    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.744799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.744704    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:50.811016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.810402    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bwr\" (UniqueName: \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.811016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.810765    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.912663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.912619    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.912847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.912740    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bwr\" (UniqueName: \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.913324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.913303    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:50.965946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:50.965910    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bwr\" (UniqueName: \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\") pod \"payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:51.025591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.025505    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:51.134496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.134461    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:09:51.416075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.416001    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:09:51.419754 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:09:51.419728    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac544671_78b1_4eb9_8cbd_9c4792aad802.slice/crio-173aed796426e3565b00a20aef255f3fb01724b4de8510883df83e8bed7565cd WatchSource:0}: Error finding container 173aed796426e3565b00a20aef255f3fb01724b4de8510883df83e8bed7565cd: Status 404 returned error can't find the container with id 173aed796426e3565b00a20aef255f3fb01724b4de8510883df83e8bed7565cd
Oct 30 22:09:51.694765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.694672    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" event={"ID":"f68ea981-4d28-4d8f-bbcc-dddf76667082","Type":"ContainerStarted","Data":"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"}
Oct 30 22:09:51.694765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.694743    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:09:51.695256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.694766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" event={"ID":"f68ea981-4d28-4d8f-bbcc-dddf76667082","Type":"ContainerStarted","Data":"10ee38c2451be6f9b243994ec43a6ccec0d0d9dc1be213c23300a7e6719beab8"}
Oct 30 22:09:51.697576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.697540    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" event={"ID":"ac544671-78b1-4eb9-8cbd-9c4792aad802","Type":"ContainerStarted","Data":"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"}
Oct 30 22:09:51.697687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.697594    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" event={"ID":"ac544671-78b1-4eb9-8cbd-9c4792aad802","Type":"ContainerStarted","Data":"173aed796426e3565b00a20aef255f3fb01724b4de8510883df83e8bed7565cd"}
Oct 30 22:09:51.697687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.697627    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:09:51.804503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.804448    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" podStartSLOduration=2.804431537 podStartE2EDuration="2.804431537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:51.802088374 +0000 UTC m=+1600.785995249" watchObservedRunningTime="2026-10-30 22:09:51.804431537 +0000 UTC m=+1600.788338413"
Oct 30 22:09:51.913215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:51.913162    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" podStartSLOduration=1.9131482819999999 podStartE2EDuration="1.913148282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:51.909520825 +0000 UTC m=+1600.893427698" watchObservedRunningTime="2026-10-30 22:09:51.913148282 +0000 UTC m=+1600.897055157"
Oct 30 22:09:53.714932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.714901    5510 generic.go:356] "Generic (PLEG): container finished" podID="544ea754-1458-4660-939e-4f974fc3cd17" containerID="d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd" exitCode=0
Oct 30 22:09:53.715353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.714974    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerDied","Data":"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"}
Oct 30 22:09:53.715398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.715379    5510 scope.go:122] "RemoveContainer" containerID="d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"
Oct 30 22:09:53.716865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.716840    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:53.716935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.716907    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:53.716975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.716940    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:09:53.957649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:53.957613    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:54.828805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:54.816297    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerStarted","Data":"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"}
Oct 30 22:09:54.998182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:54.990119    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:09:55.013465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.013430    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.013612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.013498    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:09:55.169851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.169752    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"]
Oct 30 22:09:55.170088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.170072    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"]
Oct 30 22:09:55.176685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.176648    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151b9b47-jbrjp"]
Oct 30 22:09:55.176954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.176936    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09941488-bzw9r"]
Oct 30 22:09:55.402466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.402436    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:55.607965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.607926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:55.769507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.769470    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:55.914247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.914165    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"]
Oct 30 22:09:55.921720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.921662    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-de9c059f-rkztv"]
Oct 30 22:09:55.930800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.930346    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:09:55.930800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.930585    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy" containerID="cri-o://0c0b7cd16205864a440db233ca326f2b38d7084df26b4d7f8da741f09fdbdb24" gracePeriod=30
Oct 30 22:09:55.994416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.993833    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:55.994416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.993928    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:09:55.999529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:55.999493    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:56.851300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:56.851270    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:56.990774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:56.990732    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:56.991202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:56.990840    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:09:57.128536 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:57.128462    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12959834-6b5c-4fef-9d48-c58f4dec0314" path="/var/lib/kubelet/pods/12959834-6b5c-4fef-9d48-c58f4dec0314/volumes"
Oct 30 22:09:57.129104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:57.129084    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161152ea-a97b-4a35-885f-fed6358839f3" path="/var/lib/kubelet/pods/161152ea-a97b-4a35-885f-fed6358839f3/volumes"
Oct 30 22:09:57.129692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:57.129667    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf16d92-b028-48ba-883c-c545d3dc64bd" path="/var/lib/kubelet/pods/8bf16d92-b028-48ba-883c-c545d3dc64bd/volumes"
Oct 30 22:09:57.990628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:57.990586    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:57.990849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:57.990655    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:09:58.399762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.399687    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:58.525337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.525306    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:09:58.531512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.531478    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.545847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.545820    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.545861    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.545953    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrxm\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546021    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546191    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546216    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546348    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546380    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.546508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.546493    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.576889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.576857    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:58.584866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.584837    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-446040-openshift-default-dockercfg-22tzf\""
Oct 30 22:09:58.635627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.635589    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:09:58.648427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648395    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648445    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648538    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648592    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648630    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648662    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrxm\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648709    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.648999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648923    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.649092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.648959    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.649422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.649393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.649563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.649541    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.649655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.649616    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.650241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.650185    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.650644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.650617    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.693849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.693292    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.700802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.699473    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:58.752137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.746920    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:59.012217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.938218    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:59.012217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:58.938289    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrxm\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\") pod \"e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") " pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:59.041015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.016510    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:59.041015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.016644    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:59.041015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.016705    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:09:59.161411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.161049    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:09:59.223278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.223211    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:59.990910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.990874    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:09:59.991111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:09:59.990940    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:10:00.182105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.182078    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:10:00.184639 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:00.184608    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cc0456_4000_4acf_bf92_6af5e5b87e65.slice/crio-da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d WatchSource:0}: Error finding container da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d: Status 404 returned error can't find the container with id da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d
Oct 30 22:10:00.187467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.187425    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:00.187563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.187513    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:00.187618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.187559    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:00.246685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.246651    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerStarted","Data":"da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d"}
Oct 30 22:10:00.414333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.414298    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:00.990442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.990044    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 22:10:00.990442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:00.990110    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 22:10:01.013511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.013469    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:01.375902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.375053    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:01.377651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.377616    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerStarted","Data":"929756a5e3b9cd713a388eac07a58f8f674f9175da24d0b581bcd994d83f021d"}
Oct 30 22:10:01.383589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.383219    5510 generic.go:356] "Generic (PLEG): container finished" podID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerID="0c0b7cd16205864a440db233ca326f2b38d7084df26b4d7f8da741f09fdbdb24" exitCode=0
Oct 30 22:10:01.383589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.383350    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerDied","Data":"0c0b7cd16205864a440db233ca326f2b38d7084df26b4d7f8da741f09fdbdb24"}
Oct 30 22:10:01.383589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.383405    5510 scope.go:122] "RemoveContainer" containerID="510132f7fd014b77c2c92fba747d84ca404fa3be52d4111a1baa89748274c963"
Oct 30 22:10:01.831428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.831072    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:10:01.863344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.863251    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886371    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886449    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886497    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886595    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886635    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886728    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.886997    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.887110    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\") pod \"df151195-cc3c-44b1-a8e3-b6487015d1f1\" (UID: \"df151195-cc3c-44b1-a8e3-b6487015d1f1\") "
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.887229    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.888286    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df151195-cc3c-44b1-a8e3-b6487015d1f1-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.888554    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:01.888813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.888766    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:01.892122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.892094    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:01.892419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.892401    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:01.911012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.908996    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:01.912760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.912705    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:01.913178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.913139    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:01.915913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.915892    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk" pod "df151195-cc3c-44b1-a8e3-b6487015d1f1" (UID: "df151195-cc3c-44b1-a8e3-b6487015d1f1"). InnerVolumeSpecName "kube-api-access-cztwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:01.975275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.975201    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podStartSLOduration=3.9751810020000002 podStartE2EDuration="3.975181002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:01.626290226 +0000 UTC m=+1610.610197102" watchObservedRunningTime="2026-10-30 22:10:01.975181002 +0000 UTC m=+1610.959087879"
Oct 30 22:10:01.993405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993366    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993405 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993401    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993415    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztwk\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-kube-api-access-cztwk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993428    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993440    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993454    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993467    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:01.993660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:01.993478    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df151195-cc3c-44b1-a8e3-b6487015d1f1-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:02.005395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.005355    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:10:02.006467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.006439    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:10:02.026849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.012995    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:10:02.166865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.164618    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:10:02.195860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.193485    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:10:02.393493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.393461    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"
Oct 30 22:10:02.394029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.393493    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z" event={"ID":"df151195-cc3c-44b1-a8e3-b6487015d1f1","Type":"ContainerDied","Data":"2fc8330025854c79bdaaaaf433f8bcd2155eb48e4019638cb81e63acf16a428f"}
Oct 30 22:10:02.394029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.393561    5510 scope.go:122] "RemoveContainer" containerID="0c0b7cd16205864a440db233ca326f2b38d7084df26b4d7f8da741f09fdbdb24"
Oct 30 22:10:02.394160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.394063    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:10:02.396108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.396084    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:10:02.722991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.722919    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:10:02.726749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.723771    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:10:02.726749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.724673    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-de9c059f-openshift-default-96ccf76d9-8dr5z"]
Oct 30 22:10:02.726749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:02.724853    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:10:03.075060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:03.074976    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"]
Oct 30 22:10:03.079225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:03.079195    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3335edf6-mvpql"]
Oct 30 22:10:03.129608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:03.129572    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913748d3-eb12-4a95-93fa-249cfe00183c" path="/var/lib/kubelet/pods/913748d3-eb12-4a95-93fa-249cfe00183c/volumes"
Oct 30 22:10:03.130239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:03.130220    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" path="/var/lib/kubelet/pods/df151195-cc3c-44b1-a8e3-b6487015d1f1/volumes"
Oct 30 22:10:04.471387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:04.471334    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:04.631795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:04.631752    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:04.791166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:04.791076    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:05.124872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:05.124761    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:05.308820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:05.303197    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:05.574661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:05.574618    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:05.808752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:05.808719    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:06.009592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:06.009557    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:06.525634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:06.525573    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:06.687291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:06.687259    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:08.378827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.378772    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:08.689013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.688190    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:08.960093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.960046    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:10:08.961488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.961460    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:08.969621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.969595    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:08.970130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.970107    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:08.970277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.970263    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:08.991332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:08.991303    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.039657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.039561    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:10:09.064800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.061125    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-446040-dockercfg-mmzsh\""
Oct 30 22:10:09.064800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.061228    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-446040\""
Oct 30 22:10:09.179647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.171074    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:10:09.180814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.180446    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:09.180814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.180480    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df151195-cc3c-44b1-a8e3-b6487015d1f1" containerName="istio-proxy"
Oct 30 22:10:09.185842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.185810    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.187430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.187404    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blft\" (UniqueName: \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.188337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.186704    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.328321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.319057    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blft\" (UniqueName: \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.328321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.319157    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.328321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.320228    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.421147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.421105    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.423934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.421928    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvq5r\" (UniqueName: \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.453461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.453427    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blft\" (UniqueName: \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\") pod \"payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.491094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.491057    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:10:09.534945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.533051    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvq5r\" (UniqueName: \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.534945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.533313    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.534945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.534239    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.590414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.590331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvq5r\" (UniqueName: \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\") pod \"payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:09.679028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.678994    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:09.744355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:09.744324    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:10.069306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.069265    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:10:10.075257 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:10.075231    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30ba86c_53ca_4897_8ddc_738820b838d7.slice/crio-96495b7b0b459942c6c37f7534967c71a6182710a225903b0192860aa477488b WatchSource:0}: Error finding container 96495b7b0b459942c6c37f7534967c71a6182710a225903b0192860aa477488b: Status 404 returned error can't find the container with id 96495b7b0b459942c6c37f7534967c71a6182710a225903b0192860aa477488b
Oct 30 22:10:10.169792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.169747    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:10:10.175532 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:10.175505    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131f3d9b_c66c_4fb0_89e0_aad437eb1768.slice/crio-8a39656ad193ae7e998e0077e6bc7d19a1202552144679e0a81cd80d9b640a00 WatchSource:0}: Error finding container 8a39656ad193ae7e998e0077e6bc7d19a1202552144679e0a81cd80d9b640a00: Status 404 returned error can't find the container with id 8a39656ad193ae7e998e0077e6bc7d19a1202552144679e0a81cd80d9b640a00
Oct 30 22:10:10.507188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.507085    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:10.700649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.700612    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" event={"ID":"f30ba86c-53ca-4897-8ddc-738820b838d7","Type":"ContainerStarted","Data":"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"}
Oct 30 22:10:10.700878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.700675    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" event={"ID":"f30ba86c-53ca-4897-8ddc-738820b838d7","Type":"ContainerStarted","Data":"96495b7b0b459942c6c37f7534967c71a6182710a225903b0192860aa477488b"}
Oct 30 22:10:10.700878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.700701    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:10.702491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.702459    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" event={"ID":"131f3d9b-c66c-4fb0-89e0-aad437eb1768","Type":"ContainerStarted","Data":"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"}
Oct 30 22:10:10.702619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.702511    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" event={"ID":"131f3d9b-c66c-4fb0-89e0-aad437eb1768","Type":"ContainerStarted","Data":"8a39656ad193ae7e998e0077e6bc7d19a1202552144679e0a81cd80d9b640a00"}
Oct 30 22:10:10.702619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.702539    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:10.809746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.809647    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" podStartSLOduration=2.809633849 podStartE2EDuration="2.809633849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:10.808288612 +0000 UTC m=+1619.792195488" watchObservedRunningTime="2026-10-30 22:10:10.809633849 +0000 UTC m=+1619.793540787"
Oct 30 22:10:10.915506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.915455    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" podStartSLOduration=1.9154392580000001 podStartE2EDuration="1.915439258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:10.914734246 +0000 UTC m=+1619.898641121" watchObservedRunningTime="2026-10-30 22:10:10.915439258 +0000 UTC m=+1619.899346137"
Oct 30 22:10:10.924152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:10.924120    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:11.083549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.083461    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:11.321803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.315312    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:10:11.333847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.327279    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.382801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.382022    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:10:11.420804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.383909    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778d39d3\""
Oct 30 22:10:11.447806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.447178    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778d39d3-dockercfg-gp7v5\""
Oct 30 22:10:11.595856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.595475    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.653814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.635949    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdw9r\" (UniqueName: \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.866862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.859900    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdw9r\" (UniqueName: \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.866862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.860063    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.866862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.861043    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:11.923631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:11.923604    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdw9r\" (UniqueName: \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\") pod \"payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:12.022250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.022215    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:12.140960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.140861    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 22:10:12.141176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.141112    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" containerID="cri-o://b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4" gracePeriod=30
Oct 30 22:10:12.147814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.147775    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:10:12.147980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.147961    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy" containerID="cri-o://6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668" gracePeriod=30
Oct 30 22:10:12.187126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.187081    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:10:12.187302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.187287    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy" containerID="cri-o://3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b" gracePeriod=30
Oct 30 22:10:12.190884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.190857    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 22:10:12.191042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.191022    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" containerID="cri-o://a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef" gracePeriod=30
Oct 30 22:10:12.203095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.203070    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:10:12.203240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.203222    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy" containerID="cri-o://902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276" gracePeriod=30
Oct 30 22:10:12.232322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.232295    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 22:10:12.232498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.232474    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" containerID="cri-o://3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f" gracePeriod=30
Oct 30 22:10:12.235975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.235953    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:10:12.236132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.236113    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy" containerID="cri-o://c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff" gracePeriod=30
Oct 30 22:10:12.242938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.242916    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:10:12.243100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.243074    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy" containerID="cri-o://fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6" gracePeriod=30
Oct 30 22:10:12.411189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.411111    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:10:12.416659 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:12.416622    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188c5016_ab8f_49cd_8bd7_95ffb1bcf6a2.slice/crio-7b459f5fffbba92138bc45e64af827eee67fb4bdc7ed8d72d299fb7a9ae8118b WatchSource:0}: Error finding container 7b459f5fffbba92138bc45e64af827eee67fb4bdc7ed8d72d299fb7a9ae8118b: Status 404 returned error can't find the container with id 7b459f5fffbba92138bc45e64af827eee67fb4bdc7ed8d72d299fb7a9ae8118b
Oct 30 22:10:12.925626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.925587    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" event={"ID":"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2","Type":"ContainerStarted","Data":"b74a057101c1b66c867c37227ef59dc68cec25e61e659697f6a0933e2a6a2bf8"}
Oct 30 22:10:12.926115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.925687    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:12.926115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:12.925707    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" event={"ID":"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2","Type":"ContainerStarted","Data":"7b459f5fffbba92138bc45e64af827eee67fb4bdc7ed8d72d299fb7a9ae8118b"}
Oct 30 22:10:13.215814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:13.215099    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" podStartSLOduration=2.2150785060000002 podStartE2EDuration="2.215078506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:13.214477879 +0000 UTC m=+1622.198384755" watchObservedRunningTime="2026-10-30 22:10:13.215078506 +0000 UTC m=+1622.198985385"
Oct 30 22:10:13.328521 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:13.328478    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:10:14.588669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:14.588631    5510 patch_prober.go:29] interesting pod/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:14.589219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:14.588700    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:14.812884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:14.812851    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:14.813045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:14.812914    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:15.238960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:15.238925    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:15.986266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:15.986230    5510 patch_prober.go:29] interesting pod/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:15.986677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:15.986298    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:17.618052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.618025    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:10:17.622872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.622847    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 22:10:17.641839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.641807    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:17.661213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.661185    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 22:10:17.684845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684507    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.684845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684565    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.684845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684700    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.684845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684727    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.684845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684802    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.685195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684874    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.685195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.684964    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.685195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685021    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.685195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685127    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.685195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685164    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685211    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685259    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685313    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685368    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685422    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685460    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685496    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\") pod \"ecfd46b8-29bf-4bb4-b149-936edecc0429\" (UID: \"ecfd46b8-29bf-4bb4-b149-936edecc0429\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.685546    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\") pod \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\" (UID: \"f73e9a15-8641-4194-8a92-bf5a8abf7a39\") "
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.686930    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.687417    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.688191    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.688446    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.688878    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.690917    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.691141    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.691685    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.692025    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.693326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.692221    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.706719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706609    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.706889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706764    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.706889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706773    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.706889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706860    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.707066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706918    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "kube-api-access-5vrmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.707066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.706992    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.707166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.707091    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k" pod "f73e9a15-8641-4194-8a92-bf5a8abf7a39" (UID: "f73e9a15-8641-4194-8a92-bf5a8abf7a39"). InnerVolumeSpecName "kube-api-access-5pk9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.709510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.709482    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo" pod "ecfd46b8-29bf-4bb4-b149-936edecc0429" (UID: "ecfd46b8-29bf-4bb4-b149-936edecc0429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.777603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.777515    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:10:17.795199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.792751    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.795199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.792964    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.799876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.799537    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.799876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.799624    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.799876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.799699    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.799876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.799734    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.801770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.795080    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.801770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.801199    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.801770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.801581    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.802225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.802183    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.803748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.802363    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.803748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.802453    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.803748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.802540    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\") pod \"537b4002-4472-4fce-8e88-bd52bf63a5e9\" (UID: \"537b4002-4472-4fce-8e88-bd52bf63a5e9\") "
Oct 30 22:10:17.803748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.803441    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804658    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804689    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804704    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804719    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804734    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804750    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.804861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804761    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.804774    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805859    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805875    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pk9k\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-kube-api-access-5pk9k\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805897    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805911    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805925    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805939    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrmh\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-kube-api-access-5vrmh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805958    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/537b4002-4472-4fce-8e88-bd52bf63a5e9-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805974    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f73e9a15-8641-4194-8a92-bf5a8abf7a39-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.805987    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806002    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806015    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806028    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806043    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f73e9a15-8641-4194-8a92-bf5a8abf7a39-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806058    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfd46b8-29bf-4bb4-b149-936edecc0429-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.806100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.806072    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfd46b8-29bf-4bb4-b149-936edecc0429-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.814123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.813766    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "kube-api-access-cj65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.814123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.813865    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.814123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.813939    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.814123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.814066    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo" pod "537b4002-4472-4fce-8e88-bd52bf63a5e9" (UID: "537b4002-4472-4fce-8e88-bd52bf63a5e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.828542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.828510    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 22:10:17.830758 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.830737    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:10:17.836080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.836060    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:10:17.840525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.840503    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907275    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907350    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907435    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907474    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907528    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907557    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907850    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.907988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.907935    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908007    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908057    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908095    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908156    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908188    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908242    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908278    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908356    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908402    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908470    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908504    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908562    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.908614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908608    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908640    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908760    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908817    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908890    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908936    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.908969    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\") pod \"e6dee765-db64-4ff6-981a-03feeda09b47\" (UID: \"e6dee765-db64-4ff6-981a-03feeda09b47\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909042    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909095    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909130    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909164    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\") pod \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\" (UID: \"2fc1feed-9b03-4f35-b899-181c7cc4c5f2\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909199    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909277    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\") pod \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\" (UID: \"311e7f7f-676b-455b-bbc4-cf3b4dd972e8\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909323    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.909351    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\") pod \"11386dbe-d738-4a3d-9881-e20b55970f61\" (UID: \"11386dbe-d738-4a3d-9881-e20b55970f61\") "
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.911183    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.911209    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj65p\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-kube-api-access-cj65p\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.911225    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.911239    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/537b4002-4472-4fce-8e88-bd52bf63a5e9-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.914312    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "kube-api-access-6tgx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.914583    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.914872    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.914947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.915152    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920344    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.920349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920351    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920562    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920592    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920745    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.920962    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921153    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921231    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921309    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921361    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921555    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.921670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921644    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.922324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.921935    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.922324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.922134    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.922606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.922579    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.924249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.924221    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.924631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.924604    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.924892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.924867    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.925587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.925553    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.926409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.926163    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.926409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.926245    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "kube-api-access-z42wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.926409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.926300    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.927213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.927175    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "kube-api-access-hdbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.927501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.927457    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "kube-api-access-cbgbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.929211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.929176    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.929558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.929382    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:17.929558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.929456    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy" pod "11386dbe-d738-4a3d-9881-e20b55970f61" (UID: "11386dbe-d738-4a3d-9881-e20b55970f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.929558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.929532    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token" pod "2fc1feed-9b03-4f35-b899-181c7cc4c5f2" (UID: "2fc1feed-9b03-4f35-b899-181c7cc4c5f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.930141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.930108    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy" pod "311e7f7f-676b-455b-bbc4-cf3b4dd972e8" (UID: "311e7f7f-676b-455b-bbc4-cf3b4dd972e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.930587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.930553    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.931182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.931146    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy" pod "e6dee765-db64-4ff6-981a-03feeda09b47" (UID: "e6dee765-db64-4ff6-981a-03feeda09b47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:17.946967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.946907    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:17.987392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.987364    5510 generic.go:356] "Generic (PLEG): container finished" podID="11386dbe-d738-4a3d-9881-e20b55970f61" containerID="3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b" exitCode=0
Oct 30 22:10:17.987555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.987413    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"
Oct 30 22:10:17.987555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.987445    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" event={"ID":"11386dbe-d738-4a3d-9881-e20b55970f61","Type":"ContainerDied","Data":"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"}
Oct 30 22:10:17.987555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.987502    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27" event={"ID":"11386dbe-d738-4a3d-9881-e20b55970f61","Type":"ContainerDied","Data":"a7ec7d7423c708bed0a915b09970a4ae80691c534662cc3946bb50130aec7d79"}
Oct 30 22:10:17.987555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.987522    5510 scope.go:122] "RemoveContainer" containerID="3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"
Oct 30 22:10:17.989171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.989148    5510 generic.go:356] "Generic (PLEG): container finished" podID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerID="fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6" exitCode=0
Oct 30 22:10:17.989289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.989270    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" event={"ID":"8c1f327b-3206-4358-ae30-e7bfef61053a","Type":"ContainerDied","Data":"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"}
Oct 30 22:10:17.989360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.989319    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c" event={"ID":"8c1f327b-3206-4358-ae30-e7bfef61053a","Type":"ContainerDied","Data":"ce51a2f7eed68ab4402ba571afa1affa313dbc3d2aafa39c9351a08ac4ab26cf"}
Oct 30 22:10:17.989422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.989384    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"
Oct 30 22:10:17.991429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.991403    5510 generic.go:356] "Generic (PLEG): container finished" podID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerID="c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff" exitCode=0
Oct 30 22:10:17.991588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.991568    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" event={"ID":"311e7f7f-676b-455b-bbc4-cf3b4dd972e8","Type":"ContainerDied","Data":"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"}
Oct 30 22:10:17.991661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.991618    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77" event={"ID":"311e7f7f-676b-455b-bbc4-cf3b4dd972e8","Type":"ContainerDied","Data":"0bcb0720992c00db73861f607e697e3052f94bb50dac9b3f2c344d4075189396"}
Oct 30 22:10:17.991726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.991688    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"
Oct 30 22:10:17.994741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.994717    5510 generic.go:356] "Generic (PLEG): container finished" podID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerID="b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4" exitCode=0
Oct 30 22:10:17.994864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.994813    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"
Oct 30 22:10:17.994864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.994845    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" event={"ID":"537b4002-4472-4fce-8e88-bd52bf63a5e9","Type":"ContainerDied","Data":"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"}
Oct 30 22:10:17.994987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:17.994887    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf" event={"ID":"537b4002-4472-4fce-8e88-bd52bf63a5e9","Type":"ContainerDied","Data":"7f68d871f32e27031a87617414d561947ab29f3461f64003b5e2aa8b18c4e623"}
Oct 30 22:10:18.013845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.013818    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.013907    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014044    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014047    5510 generic.go:356] "Generic (PLEG): container finished" podID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerID="3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f" exitCode=0
Oct 30 22:10:18.014171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014134    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014170    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:18.014341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014191    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"
Oct 30 22:10:18.014341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014233    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014291    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014330    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerDied","Data":"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"}
Oct 30 22:10:18.014583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014373    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4" event={"ID":"2fc1feed-9b03-4f35-b899-181c7cc4c5f2","Type":"ContainerDied","Data":"b0bbc6deb653b6a43e3035e156e6c54adbdc0b0ca72fe93c8f2caf85f83678c0"}
Oct 30 22:10:18.014583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014382    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:18.014687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014581    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:18.014687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014604    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:18.014687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014649    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.014911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.014693    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\") pod \"8c1f327b-3206-4358-ae30-e7bfef61053a\" (UID: \"8c1f327b-3206-4358-ae30-e7bfef61053a\") "
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015396    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgbn\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-kube-api-access-cbgbn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015426    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015440    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015453    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015468    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015482    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015494    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015507    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015521    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dee765-db64-4ff6-981a-03feeda09b47-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015533    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015546    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015559    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42wn\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-kube-api-access-z42wn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015572    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11386dbe-d738-4a3d-9881-e20b55970f61-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015584    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015599    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015612    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015626    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015630    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015640    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11386dbe-d738-4a3d-9881-e20b55970f61-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015677    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015696    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015712    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgx2\" (UniqueName: \"kubernetes.io/projected/e6dee765-db64-4ff6-981a-03feeda09b47-kube-api-access-6tgx2\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015725    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dee765-db64-4ff6-981a-03feeda09b47-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015740    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015755    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015769    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015825    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015842    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015856    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015868    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015881    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11386dbe-d738-4a3d-9881-e20b55970f61-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015895    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015911    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015927    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015940    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dee765-db64-4ff6-981a-03feeda09b47-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015957    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11386dbe-d738-4a3d-9881-e20b55970f61-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015970    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015983    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.015997    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.016013    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc1feed-9b03-4f35-b899-181c7cc4c5f2-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.024750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.016029    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbgr\" (UniqueName: \"kubernetes.io/projected/311e7f7f-676b-455b-bbc4-cf3b4dd972e8-kube-api-access-hdbgr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.038061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.036364    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "kube-api-access-l7q7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:18.038061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.036903    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:18.039449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.039419    5510 scope.go:122] "RemoveContainer" containerID="3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"
Oct 30 22:10:18.039723 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.039706    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b\": container with ID starting with 3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b not found: ID does not exist" containerID="3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"
Oct 30 22:10:18.039836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.039732    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b"} err="failed to get container status \"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b\": rpc error: code = NotFound desc = could not find container \"3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b\": container with ID starting with 3ac10e137588c3d2613277a6cf62076615fe2932fc2877eb1de24d0fe0d3816b not found: ID does not exist"
Oct 30 22:10:18.039836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.039752    5510 scope.go:122] "RemoveContainer" containerID="fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"
Oct 30 22:10:18.040472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.040444    5510 generic.go:356] "Generic (PLEG): container finished" podID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerID="6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668" exitCode=0
Oct 30 22:10:18.040585 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.040545    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" event={"ID":"f73e9a15-8641-4194-8a92-bf5a8abf7a39","Type":"ContainerDied","Data":"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"}
Oct 30 22:10:18.040661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.040594    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9" event={"ID":"f73e9a15-8641-4194-8a92-bf5a8abf7a39","Type":"ContainerDied","Data":"67cea27bc0934f859674d2807d691d0facfd76273ef6697c5be95a4642e022ef"}
Oct 30 22:10:18.040920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.040900    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"
Oct 30 22:10:18.041561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.041540    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:18.041651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.041628    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo" pod "8c1f327b-3206-4358-ae30-e7bfef61053a" (UID: "8c1f327b-3206-4358-ae30-e7bfef61053a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:18.043007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.042982    5510 generic.go:356] "Generic (PLEG): container finished" podID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerID="a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef" exitCode=0
Oct 30 22:10:18.043112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.043013    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" event={"ID":"ecfd46b8-29bf-4bb4-b149-936edecc0429","Type":"ContainerDied","Data":"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"}
Oct 30 22:10:18.043112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.043064    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"
Oct 30 22:10:18.043112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.043072    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6" event={"ID":"ecfd46b8-29bf-4bb4-b149-936edecc0429","Type":"ContainerDied","Data":"d8455a32cc821a772b5ed6a4a1678329bb4f18923a74ca6844a3bffe67f89a7c"}
Oct 30 22:10:18.044903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.044848    5510 generic.go:356] "Generic (PLEG): container finished" podID="e6dee765-db64-4ff6-981a-03feeda09b47" containerID="902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276" exitCode=0
Oct 30 22:10:18.044903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.044873    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" event={"ID":"e6dee765-db64-4ff6-981a-03feeda09b47","Type":"ContainerDied","Data":"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"}
Oct 30 22:10:18.045064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.044910    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"
Oct 30 22:10:18.045064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.044930    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4" event={"ID":"e6dee765-db64-4ff6-981a-03feeda09b47","Type":"ContainerDied","Data":"96e9c08655b72b1ea766a73f6ead4fdab764c690803d96f788edf764c55aaca7"}
Oct 30 22:10:18.087048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.087025    5510 scope.go:122] "RemoveContainer" containerID="fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"
Oct 30 22:10:18.087416 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.087379    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6\": container with ID starting with fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6 not found: ID does not exist" containerID="fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"
Oct 30 22:10:18.087548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.087413    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6"} err="failed to get container status \"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6\": rpc error: code = NotFound desc = could not find container \"fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6\": container with ID starting with fd7c381bb2ad7104ca52a2492d97e4ee22050174c055329402853d0636abfdd6 not found: ID does not exist"
Oct 30 22:10:18.087548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.087436    5510 scope.go:122] "RemoveContainer" containerID="c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"
Oct 30 22:10:18.098749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.098729    5510 scope.go:122] "RemoveContainer" containerID="c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"
Oct 30 22:10:18.099071 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.099036    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff\": container with ID starting with c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff not found: ID does not exist" containerID="c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"
Oct 30 22:10:18.099171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.099066    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff"} err="failed to get container status \"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff\": rpc error: code = NotFound desc = could not find container \"c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff\": container with ID starting with c444455ce6e6a69d0a20535fe38e43021194b2444d184e75b0a54e5ef08775ff not found: ID does not exist"
Oct 30 22:10:18.099171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.099089    5510 scope.go:122] "RemoveContainer" containerID="b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"
Oct 30 22:10:18.110937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.110905    5510 scope.go:122] "RemoveContainer" containerID="b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"
Oct 30 22:10:18.111209 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.111189    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4\": container with ID starting with b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4 not found: ID does not exist" containerID="b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"
Oct 30 22:10:18.111285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.111214    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4"} err="failed to get container status \"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4\": rpc error: code = NotFound desc = could not find container \"b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4\": container with ID starting with b8a4f2cc516a23fd31d581d076383199ad8e8d24b89eaf12aeb48819c32f6ee4 not found: ID does not exist"
Oct 30 22:10:18.111285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.111230    5510 scope.go:122] "RemoveContainer" containerID="3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"
Oct 30 22:10:18.117478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.117454    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.117571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.117482    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7q7x\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-kube-api-access-l7q7x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.117571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.117500    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1f327b-3206-4358-ae30-e7bfef61053a-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.117571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.117513    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.117571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.117526    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1f327b-3206-4358-ae30-e7bfef61053a-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.120925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.120894    5510 scope.go:122] "RemoveContainer" containerID="b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"
Oct 30 22:10:18.131162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.131139    5510 scope.go:122] "RemoveContainer" containerID="3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"
Oct 30 22:10:18.131596 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.131565    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f\": container with ID starting with 3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f not found: ID does not exist" containerID="3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"
Oct 30 22:10:18.131596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.131605    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f"} err="failed to get container status \"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f\": rpc error: code = NotFound desc = could not find container \"3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f\": container with ID starting with 3389f395547986c27145e17386384b51d54e382363e55ce67d7fd4544b3e510f not found: ID does not exist"
Oct 30 22:10:18.131596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.131683    5510 scope.go:122] "RemoveContainer" containerID="b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"
Oct 30 22:10:18.132285 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.132029    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff\": container with ID starting with b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff not found: ID does not exist" containerID="b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"
Oct 30 22:10:18.132285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.132057    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff"} err="failed to get container status \"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff\": rpc error: code = NotFound desc = could not find container \"b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff\": container with ID starting with b6338c4b1b986d1a063700466c95b1c376245ce6a42feb8bfb26fb856dcd86ff not found: ID does not exist"
Oct 30 22:10:18.132285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.132075    5510 scope.go:122] "RemoveContainer" containerID="6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"
Oct 30 22:10:18.157318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.150710    5510 scope.go:122] "RemoveContainer" containerID="6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"
Oct 30 22:10:18.157318 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.151075    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668\": container with ID starting with 6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668 not found: ID does not exist" containerID="6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"
Oct 30 22:10:18.157318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.151111    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668"} err="failed to get container status \"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668\": rpc error: code = NotFound desc = could not find container \"6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668\": container with ID starting with 6d5801e4abc46ff5a1cf28c02115bccac49ecb0945343e0b996ebdf3b2dc1668 not found: ID does not exist"
Oct 30 22:10:18.157318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.151140    5510 scope.go:122] "RemoveContainer" containerID="a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"
Oct 30 22:10:18.248965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.248941    5510 scope.go:122] "RemoveContainer" containerID="a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"
Oct 30 22:10:18.249313 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.249291    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef\": container with ID starting with a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef not found: ID does not exist" containerID="a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"
Oct 30 22:10:18.249366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.249325    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef"} err="failed to get container status \"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef\": rpc error: code = NotFound desc = could not find container \"a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef\": container with ID starting with a20246bc5fb977360446862af7853dbca293c44ca191e9e8166bbbd5b47654ef not found: ID does not exist"
Oct 30 22:10:18.249366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.249344    5510 scope.go:122] "RemoveContainer" containerID="902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"
Oct 30 22:10:18.260134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.260106    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 22:10:18.266181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.266080    5510 scope.go:122] "RemoveContainer" containerID="902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"
Oct 30 22:10:18.277806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.270733    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-gbnkf"]
Oct 30 22:10:18.278108 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:18.278025    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276\": container with ID starting with 902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276 not found: ID does not exist" containerID="902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"
Oct 30 22:10:18.278108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.278076    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276"} err="failed to get container status \"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276\": rpc error: code = NotFound desc = could not find container \"902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276\": container with ID starting with 902934f0cc941db2de603d8b31b1f7a424f808a3a6d5cf589cb76393469aa276 not found: ID does not exist"
Oct 30 22:10:18.562890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.562801    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:10:18.569174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.569149    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-rhn77"]
Oct 30 22:10:18.670679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.670642    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:10:18.676342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.676307    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-kk8v9"]
Oct 30 22:10:18.776306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.776263    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:10:18.780017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.779994    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-fvk27"]
Oct 30 22:10:18.947492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.947419    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 22:10:18.953999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:18.953970    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-kn8n4"]
Oct 30 22:10:19.308805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.303556    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" path="/var/lib/kubelet/pods/11386dbe-d738-4a3d-9881-e20b55970f61/volumes"
Oct 30 22:10:19.308805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.304557    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" path="/var/lib/kubelet/pods/2fc1feed-9b03-4f35-b899-181c7cc4c5f2/volumes"
Oct 30 22:10:19.308805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.305336    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" path="/var/lib/kubelet/pods/311e7f7f-676b-455b-bbc4-cf3b4dd972e8/volumes"
Oct 30 22:10:19.328597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.324619    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" path="/var/lib/kubelet/pods/537b4002-4472-4fce-8e88-bd52bf63a5e9/volumes"
Oct 30 22:10:19.334268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.334234    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" path="/var/lib/kubelet/pods/f73e9a15-8641-4194-8a92-bf5a8abf7a39/volumes"
Oct 30 22:10:19.335014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.334988    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:10:19.397522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.397479    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-hk8p4"]
Oct 30 22:10:19.397724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.397537    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 22:10:19.397724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.397563    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f776cb-openshift-default-74d577586f-xx6s6"]
Oct 30 22:10:19.490413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.490376    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:10:19.496038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.496006    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5be0d1-openshift-default-55cd484f99-shn2c"]
Oct 30 22:10:19.821107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:19.821073    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:21.127128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.127048    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" path="/var/lib/kubelet/pods/8c1f327b-3206-4358-ae30-e7bfef61053a/volumes"
Oct 30 22:10:21.127587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.127563    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" path="/var/lib/kubelet/pods/e6dee765-db64-4ff6-981a-03feeda09b47/volumes"
Oct 30 22:10:21.128039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.128024    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" path="/var/lib/kubelet/pods/ecfd46b8-29bf-4bb4-b149-936edecc0429/volumes"
Oct 30 22:10:21.637091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.637051    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:21.918923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.918812    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:10:21.919129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:21.919107    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:10:22.076503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.074021    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:22.281506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.281468    5510 scope.go:122] "RemoveContainer" containerID="1651c9d74391fe9ae32abdd1db1c50463740278d1d3e759429d48cf89c4b0a88"
Oct 30 22:10:22.294401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.294348    5510 scope.go:122] "RemoveContainer" containerID="a6cab293f0011faec2fada4317d080d3f98b3bd73ce097780008961fb97fcc0a"
Oct 30 22:10:22.307834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.307811    5510 scope.go:122] "RemoveContainer" containerID="bae9d3f438874fc6a9adf66e3c7d6a203caac979be8b0b6e0b184fb164d1439a"
Oct 30 22:10:22.318068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.318050    5510 scope.go:122] "RemoveContainer" containerID="6c21b15c90caf39e852fa6e990ef27560e59c152ce153be7a503641334500a37"
Oct 30 22:10:22.327528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.327508    5510 scope.go:122] "RemoveContainer" containerID="433777bb164149ebe8c5ef47104fdfb7445845d858b1160e806e7c008bdac23b"
Oct 30 22:10:22.336667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.336647    5510 scope.go:122] "RemoveContainer" containerID="347c431b3fa97df8064168f4accf36c10fc2029374ca5c15d8083fdbf5bf1cd7"
Oct 30 22:10:22.346023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.346000    5510 scope.go:122] "RemoveContainer" containerID="90b00fbd6ac970844fae06f4061b6224c3b9ccc092e4648c9a71decf88d55e00"
Oct 30 22:10:22.356402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.356382    5510 scope.go:122] "RemoveContainer" containerID="e2cfd80390b0558b979a8473a1123f96863bdd162d74edab42fcf130e9f10a15"
Oct 30 22:10:22.367499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.367474    5510 scope.go:122] "RemoveContainer" containerID="531153a5796c8728614388c8c6d04ef929ddd6684b0759914e17c3f62288017d"
Oct 30 22:10:22.722803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.722255    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723300    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723325    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723345    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723353    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723373    5510 cpu_manager.go:414] "containerMap: removing container" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723381    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723396    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723402    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723414    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723420    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723435    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723444    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723461    5510 cpu_manager.go:414] "containerMap: removing container" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723468    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723509    5510 cpu_manager.go:414] "containerMap: removing container" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723516    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723533    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723540    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723691    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="11386dbe-d738-4a3d-9881-e20b55970f61" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723710    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dee765-db64-4ff6-981a-03feeda09b47" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723724    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfd46b8-29bf-4bb4-b149-936edecc0429" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723734    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723746    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="537b4002-4472-4fce-8e88-bd52bf63a5e9" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723757    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73e9a15-8641-4194-8a92-bf5a8abf7a39" containerName="istio-proxy"
Oct 30 22:10:22.733803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.723768    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc1feed-9b03-4f35-b899-181c7cc4c5f2" containerName="istio-proxy"
Oct 30 22:10:22.864885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.861658    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:22.911526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.897876    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1f327b-3206-4358-ae30-e7bfef61053a" containerName="istio-proxy"
Oct 30 22:10:22.911526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.897948    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="311e7f7f-676b-455b-bbc4-cf3b4dd972e8" containerName="istio-proxy"
Oct 30 22:10:22.942812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.921559    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:10:22.942812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.921687    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:22.980713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:22.979941    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-02bd3b-openshift-default-dockercfg-vvkbp\""
Oct 30 22:10:23.071767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.071723    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072075    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072281    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pj69\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072414    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072547    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072799    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.072946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.072933    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.073054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.073042    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.073219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.073204    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.189320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.189232    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pj69\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.189540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.189522    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.189751 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.189728    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.203093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.203062    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.203293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.203277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.203416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.203403    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.203592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.203575    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.203726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.203713    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.207733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.207708    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.214109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.190913    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.214109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.206372    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.214109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.207054    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.214109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.206164    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.214906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.214877    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.229741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.221559    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:23.230038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.229574    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.230038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.229840    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.258612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.258576    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.259697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.259673    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pj69\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\") pod \"e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") " pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.345536 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.345508    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:23.794604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.794559    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:10:23.826863 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:23.826832    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d015c53_755f_4af5_a5b0_cce8bf55ba00.slice/crio-8a3a51e6349a2af38b35fec131dffd4e6c4685d2d7f0e860408c09c9c5b67432 WatchSource:0}: Error finding container 8a3a51e6349a2af38b35fec131dffd4e6c4685d2d7f0e860408c09c9c5b67432: Status 404 returned error can't find the container with id 8a3a51e6349a2af38b35fec131dffd4e6c4685d2d7f0e860408c09c9c5b67432
Oct 30 22:10:23.829466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.829432    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:23.829534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.829504    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:23.829567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.829532    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:23.938596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:23.938567    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:10:24.526840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:24.526792    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerStarted","Data":"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"}
Oct 30 22:10:24.527442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:24.527411    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerStarted","Data":"8a3a51e6349a2af38b35fec131dffd4e6c4685d2d7f0e860408c09c9c5b67432"}
Oct 30 22:10:24.648443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:24.648382    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podStartSLOduration=2.6483635210000003 podStartE2EDuration="2.648363521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:24.644882301 +0000 UTC m=+1633.628789178" watchObservedRunningTime="2026-10-30 22:10:24.648363521 +0000 UTC m=+1633.632270399"
Oct 30 22:10:24.694517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:24.694481    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:10:24.860718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:24.860587    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:25.345930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:25.345896    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:25.352491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:25.352462    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:25.534849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:25.534816    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:25.535926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:25.535902    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:10:26.175583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:26.175547    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:26.375884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:26.375853    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:29.387172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:29.387132    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:30.046380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:30.046346    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"]
Oct 30 22:10:30.092657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:30.092624    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea30ba4d-xnwdc"]
Oct 30 22:10:31.126798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:31.126744    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58878a49-5c1c-4736-aac3-9a27a4b0694e" path="/var/lib/kubelet/pods/58878a49-5c1c-4736-aac3-9a27a4b0694e/volumes"
Oct 30 22:10:33.417762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:33.417729    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:33.658754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:33.658723    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:33.823227 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:33.823191    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:33.977702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:33.977665    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:34.007798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.007749    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:10:34.012308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.012287    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.116860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.115773    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:10:34.116860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.116738    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.116860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.116817    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7wh\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.116983    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117011    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117048    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117079    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117183    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117256    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.117448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.117334    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.143803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.133513    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-6ea8da-openshift-default-dockercfg-l7qd7\""
Oct 30 22:10:34.185815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.184421    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438183    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438259    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438338    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438396    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438634    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.438915    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.439163    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.439254    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.439277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7wh\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.440118    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.440598    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.442330    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.443864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.442434    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.454756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.452959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.495801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.488089    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.538221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.538184    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:34.554812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.549379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.561920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.555871    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7wh\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.577719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.577627    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\") pod \"e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") " pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:34.672633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:34.671446    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:35.145843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.145737    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:10:35.149670 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:35.149648    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f360d1_daf8_45e0_9c66_71327d4f5b28.slice/crio-18b05f190cf8bc5db99aaa258e0bd96a52330bf6c74962a55f38fe100d26aaf4 WatchSource:0}: Error finding container 18b05f190cf8bc5db99aaa258e0bd96a52330bf6c74962a55f38fe100d26aaf4: Status 404 returned error can't find the container with id 18b05f190cf8bc5db99aaa258e0bd96a52330bf6c74962a55f38fe100d26aaf4
Oct 30 22:10:35.152256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.152220    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:35.152384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.152285    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:35.152384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.152312    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:10:35.625747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.625706    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerStarted","Data":"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"}
Oct 30 22:10:35.626511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.625821    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerStarted","Data":"18b05f190cf8bc5db99aaa258e0bd96a52330bf6c74962a55f38fe100d26aaf4"}
Oct 30 22:10:35.672619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.672590    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:35.781931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:35.781869    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podStartSLOduration=2.781852363 podStartE2EDuration="2.781852363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:35.781028391 +0000 UTC m=+1644.764935267" watchObservedRunningTime="2026-10-30 22:10:35.781852363 +0000 UTC m=+1644.765759241"
Oct 30 22:10:36.679363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:36.679328    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:37.565470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:37.565428    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:37.641112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:37.641082    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:37.642515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:37.642489    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:10:37.740302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:37.740273    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:37.905264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:37.905189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:38.644483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:38.644444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:39.423446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:39.423417    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:39.600559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:39.600528    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:39.764885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:39.764844    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:39.924278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:39.924245    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:41.684133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:41.684097    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:41.855283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:41.855252    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:42.098321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.098276    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:10:42.134062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.122683    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.251755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.251729    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-02bd3b\""
Oct 30 22:10:42.345282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.345252    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt84\" (UniqueName: \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.345881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.345849    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.359703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.359624    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:10:42.373218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.373177    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:10:42.418114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.418075    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.451399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.449694    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.451399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.449930    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt84\" (UniqueName: \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.451399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.451347    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.510871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.491452    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-02bd3b-dockercfg-wblvf\""
Oct 30 22:10:42.558249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.552083    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.558249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.553774    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg8b\" (UniqueName: \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.602618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.568012    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:10:42.604566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.586739    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt84\" (UniqueName: \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\") pod \"payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.604714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.604520    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:10:42.604714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.604683    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 22:10:42.606715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.604938    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" containerID="cri-o://bfbeda8cdc5826c9c3d15bdb0b185d149698b65a9d728f3d37d5238297bfc67f" gracePeriod=30
Oct 30 22:10:42.606715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.605186    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy" containerID="cri-o://5dbedc6caec1c18ba3a045f41618ab25d39c16b00e36057f393e916205662c96" gracePeriod=30
Oct 30 22:10:42.606715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.605450    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" containerID="cri-o://c91010de94c646711db7df6a37cea71ec2fb4cf2916a5bba1900004381518770" gracePeriod=30
Oct 30 22:10:42.607534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.606972    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:42.612011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.611584    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:10:42.655439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.655404    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.655599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.655457    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg8b\" (UniqueName: \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.656362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.656334    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.711448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.711422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg8b\" (UniqueName: \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\") pod \"payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.912226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.912145    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:42.995264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:42.994915    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:10:42.998394 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:42.998361    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2dbd35_2da5_40a3_9818_79c5428801f2.slice/crio-8c52719790633730001e5adece6f318aa9cd85d38ba87a7f837a641f6816ff5c WatchSource:0}: Error finding container 8c52719790633730001e5adece6f318aa9cd85d38ba87a7f837a641f6816ff5c: Status 404 returned error can't find the container with id 8c52719790633730001e5adece6f318aa9cd85d38ba87a7f837a641f6816ff5c
Oct 30 22:10:43.285355 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:10:43.285320    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:10:43.311432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.311401    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:10:43.317611 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:43.317579    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbe2544_9324_459c_b40f_5c6c95aff125.slice/crio-6bacd684bed2f304e3e31fa6ee4d84a82e6d36fdabebab9d693055f5b02d2657 WatchSource:0}: Error finding container 6bacd684bed2f304e3e31fa6ee4d84a82e6d36fdabebab9d693055f5b02d2657: Status 404 returned error can't find the container with id 6bacd684bed2f304e3e31fa6ee4d84a82e6d36fdabebab9d693055f5b02d2657
Oct 30 22:10:43.692993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.692894    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" event={"ID":"0cbe2544-9324-459c-b40f-5c6c95aff125","Type":"ContainerStarted","Data":"d2db65c8efbdae32213da521f7c8a0102aa7ed90ff7b9651383d181e1dc4d89e"}
Oct 30 22:10:43.692993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.692961    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" event={"ID":"0cbe2544-9324-459c-b40f-5c6c95aff125","Type":"ContainerStarted","Data":"6bacd684bed2f304e3e31fa6ee4d84a82e6d36fdabebab9d693055f5b02d2657"}
Oct 30 22:10:43.693235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.693003    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:43.694639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.694608    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" event={"ID":"df2dbd35-2da5-40a3-9818-79c5428801f2","Type":"ContainerStarted","Data":"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"}
Oct 30 22:10:43.694763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.694664    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" event={"ID":"df2dbd35-2da5-40a3-9818-79c5428801f2","Type":"ContainerStarted","Data":"8c52719790633730001e5adece6f318aa9cd85d38ba87a7f837a641f6816ff5c"}
Oct 30 22:10:43.694763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.694694    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:43.755201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.755173    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:43.813808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:43.812372    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" podStartSLOduration=1.8123516450000001 podStartE2EDuration="1.812351645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:43.804347948 +0000 UTC m=+1652.788254836" watchObservedRunningTime="2026-10-30 22:10:43.812351645 +0000 UTC m=+1652.796258516"
Oct 30 22:10:45.663149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:45.663117    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:45.802960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:45.802924    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:45.803104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:45.802980    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:46.459883 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.459849    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:46.762855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.726098    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:46.777010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.776901    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:46.777365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.777317    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:46.860805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.859848    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" podStartSLOduration=4.859826773 podStartE2EDuration="4.859826773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:43.928745165 +0000 UTC m=+1652.912652041" watchObservedRunningTime="2026-10-30 22:10:46.859826773 +0000 UTC m=+1655.843733650"
Oct 30 22:10:46.919231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.908896    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:10:46.919231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.917499    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:46.938802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.932855    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:46.938802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.932997    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6f4p\" (UniqueName: \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:46.938802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.937151    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:10:46.972805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.972110    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-6ea8da-dockercfg-r65nx\""
Oct 30 22:10:46.972805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:46.972237    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-6ea8da\""
Oct 30 22:10:47.153889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.084150    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:47.153889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.084257    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6f4p\" (UniqueName: \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:47.153889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.085557    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:47.241802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.241570    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6f4p\" (UniqueName: \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\") pod \"payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:47.290949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.290914    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:10:47.383306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.383275    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:10:47.404528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.404447    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.463537 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.463505    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:47.489385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.489119    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxmx\" (UniqueName: \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.489616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.489584    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.591075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.591041    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.591260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.591203    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxmx\" (UniqueName: \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.592004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.591972    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.644459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.644435    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxmx\" (UniqueName: \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\") pod \"payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.754158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.754114    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:47.822859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.822125    5510 generic.go:356] "Generic (PLEG): container finished" podID="61efb1f7-04b2-49bc-a633-de94b399353f" containerID="c91010de94c646711db7df6a37cea71ec2fb4cf2916a5bba1900004381518770" exitCode=0
Oct 30 22:10:47.822859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.822318    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" event={"ID":"61efb1f7-04b2-49bc-a633-de94b399353f","Type":"ContainerDied","Data":"c91010de94c646711db7df6a37cea71ec2fb4cf2916a5bba1900004381518770"}
Oct 30 22:10:47.825281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.824564    5510 generic.go:356] "Generic (PLEG): container finished" podID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerID="bfbeda8cdc5826c9c3d15bdb0b185d149698b65a9d728f3d37d5238297bfc67f" exitCode=0
Oct 30 22:10:47.825281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.824816    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" event={"ID":"10b5f0a4-9839-4721-841c-0cc6f7139c54","Type":"ContainerDied","Data":"bfbeda8cdc5826c9c3d15bdb0b185d149698b65a9d728f3d37d5238297bfc67f"}
Oct 30 22:10:47.829798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.829022    5510 generic.go:356] "Generic (PLEG): container finished" podID="d93f619e-ee34-4897-a528-9a6971e23e24" containerID="5dbedc6caec1c18ba3a045f41618ab25d39c16b00e36057f393e916205662c96" exitCode=0
Oct 30 22:10:47.829798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.829080    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" event={"ID":"d93f619e-ee34-4897-a528-9a6971e23e24","Type":"ContainerDied","Data":"5dbedc6caec1c18ba3a045f41618ab25d39c16b00e36057f393e916205662c96"}
Oct 30 22:10:47.853330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.853295    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:10:47.923702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.923546    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:10:47.955671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:47.954946    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:10:48.008279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008238    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008336    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008363    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008411    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008442    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008456    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008475    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008502    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008531    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008598    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008623    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.008657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.009148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008755    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.009148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.008833    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.009299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.009278    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.009587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.009554    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.009695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.009669    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\") pod \"10b5f0a4-9839-4721-841c-0cc6f7139c54\" (UID: \"10b5f0a4-9839-4721-841c-0cc6f7139c54\") "
Oct 30 22:10:48.009772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.009758    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\") pod \"d93f619e-ee34-4897-a528-9a6971e23e24\" (UID: \"d93f619e-ee34-4897-a528-9a6971e23e24\") "
Oct 30 22:10:48.009873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.009845    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.010549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.010521    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:48.010652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.010588    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.010652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.010615    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.010652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.010631    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.010936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.010906    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.011154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.011132    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.011233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.011200    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.011298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.011255    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.011425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.011404    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:48.011635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.011614    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.013196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.013175    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.013866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.013744    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:48.014529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.014502    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.014854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.014828    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "kube-api-access-9f9mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.015321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.015293    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:48.015429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.015314    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.015429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.015351    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978" pod "d93f619e-ee34-4897-a528-9a6971e23e24" (UID: "d93f619e-ee34-4897-a528-9a6971e23e24"). InnerVolumeSpecName "kube-api-access-2x978". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.016934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.016903    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token" pod "10b5f0a4-9839-4721-841c-0cc6f7139c54" (UID: "10b5f0a4-9839-4721-841c-0cc6f7139c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.023202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.023183    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 22:10:48.111664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111629    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.111854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111691    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.111854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111767    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.111989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111883    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.111989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111967    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.112103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.111989    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.112103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112011    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.112103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112040    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.112103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.112319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112105    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\") pod \"61efb1f7-04b2-49bc-a633-de94b399353f\" (UID: \"61efb1f7-04b2-49bc-a633-de94b399353f\") "
Oct 30 22:10:48.112319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112140    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.112434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112404    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:48.112756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112701    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.112756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112723    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112797    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x978\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-kube-api-access-2x978\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112823    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112839    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b5f0a4-9839-4721-841c-0cc6f7139c54-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112854    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112871    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112887    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112903    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112917    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112932    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.112958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112948    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112963    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f9mb\" (UniqueName: \"kubernetes.io/projected/10b5f0a4-9839-4721-841c-0cc6f7139c54-kube-api-access-9f9mb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112977    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.112991    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efb1f7-04b2-49bc-a633-de94b399353f-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.113004    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93f619e-ee34-4897-a528-9a6971e23e24-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.113019    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93f619e-ee34-4897-a528-9a6971e23e24-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.113034    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93f619e-ee34-4897-a528-9a6971e23e24-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.113052    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b5f0a4-9839-4721-841c-0cc6f7139c54-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.113475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.113066    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93f619e-ee34-4897-a528-9a6971e23e24-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.114183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.114154    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.114272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.114192    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:48.114399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.114384    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.114497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.114478    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m" pod "61efb1f7-04b2-49bc-a633-de94b399353f" (UID: "61efb1f7-04b2-49bc-a633-de94b399353f"). InnerVolumeSpecName "kube-api-access-64b4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.174484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.174274    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:10:48.178052 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:10:48.178006    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8005249_6108_446f_a8f5_5e3fe342e034.slice/crio-59e2055cb0c25911adf1320a816e55a4360f72558c9486435984fa57ef477d33 WatchSource:0}: Error finding container 59e2055cb0c25911adf1320a816e55a4360f72558c9486435984fa57ef477d33: Status 404 returned error can't find the container with id 59e2055cb0c25911adf1320a816e55a4360f72558c9486435984fa57ef477d33
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214818    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64b4m\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-kube-api-access-64b4m\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214848    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214864    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214878    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efb1f7-04b2-49bc-a633-de94b399353f-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214893    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efb1f7-04b2-49bc-a633-de94b399353f-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.215129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.214906    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efb1f7-04b2-49bc-a633-de94b399353f-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.655869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.655829    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:48.848055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.848022    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"
Oct 30 22:10:48.848055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.848046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz" event={"ID":"10b5f0a4-9839-4721-841c-0cc6f7139c54","Type":"ContainerDied","Data":"df397a49f3707e6cb0f3bd376fea531e8e62a702ed1f14af88fd78f3ebe0093c"}
Oct 30 22:10:48.848528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.848142    5510 scope.go:122] "RemoveContainer" containerID="bfbeda8cdc5826c9c3d15bdb0b185d149698b65a9d728f3d37d5238297bfc67f"
Oct 30 22:10:48.850046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.850020    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" event={"ID":"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9","Type":"ContainerStarted","Data":"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"}
Oct 30 22:10:48.850182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.850075    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:10:48.850182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.850098    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" event={"ID":"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9","Type":"ContainerStarted","Data":"c8f58018a2ea7d8528f192b04de42079c8f6a447b9bce69eea08d047ed12aed8"}
Oct 30 22:10:48.851616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.851593    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"
Oct 30 22:10:48.851616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.851595    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph" event={"ID":"d93f619e-ee34-4897-a528-9a6971e23e24","Type":"ContainerDied","Data":"a6fc24813fb3f92388b9579baf4f13c7a8251380afd6a1be5a359878fb627aeb"}
Oct 30 22:10:48.853505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.853397    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" event={"ID":"d8005249-6108-446f-a8f5-5e3fe342e034","Type":"ContainerStarted","Data":"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"}
Oct 30 22:10:48.853505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.853445    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" event={"ID":"d8005249-6108-446f-a8f5-5e3fe342e034","Type":"ContainerStarted","Data":"59e2055cb0c25911adf1320a816e55a4360f72558c9486435984fa57ef477d33"}
Oct 30 22:10:48.853663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.853518    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:48.855081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.855060    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"
Oct 30 22:10:48.855242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.855116    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g" event={"ID":"61efb1f7-04b2-49bc-a633-de94b399353f","Type":"ContainerDied","Data":"8a56936509280911535ec036d6e79d83115037dc7bc31baeccfb90f380353d90"}
Oct 30 22:10:48.862541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.862523    5510 scope.go:122] "RemoveContainer" containerID="5dbedc6caec1c18ba3a045f41618ab25d39c16b00e36057f393e916205662c96"
Oct 30 22:10:48.875709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:48.875688    5510 scope.go:122] "RemoveContainer" containerID="c91010de94c646711db7df6a37cea71ec2fb4cf2916a5bba1900004381518770"
Oct 30 22:10:49.002514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.002484    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 22:10:49.006177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.006139    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-ssp5g"]
Oct 30 22:10:49.114031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.113975    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" podStartSLOduration=3.113961982 podStartE2EDuration="3.113961982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.110362872 +0000 UTC m=+1658.094269748" watchObservedRunningTime="2026-10-30 22:10:49.113961982 +0000 UTC m=+1658.097868896"
Oct 30 22:10:49.132863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.132831    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" path="/var/lib/kubelet/pods/61efb1f7-04b2-49bc-a633-de94b399353f/volumes"
Oct 30 22:10:49.218505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.217944    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:10:49.233817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.225294    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-g9vph"]
Oct 30 22:10:49.350884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.348968    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" podStartSLOduration=3.348949358 podStartE2EDuration="3.348949358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.337792868 +0000 UTC m=+1658.321699737" watchObservedRunningTime="2026-10-30 22:10:49.348949358 +0000 UTC m=+1658.332856235"
Oct 30 22:10:49.442886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.442850    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:10:49.447767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:49.447740    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-qk4mz"]
Oct 30 22:10:51.126773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:51.126738    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" path="/var/lib/kubelet/pods/10b5f0a4-9839-4721-841c-0cc6f7139c54/volumes"
Oct 30 22:10:51.127332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:51.127316    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" path="/var/lib/kubelet/pods/d93f619e-ee34-4897-a528-9a6971e23e24/volumes"
Oct 30 22:10:53.456252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:53.456189    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:54.705431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:54.705399    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:10:54.705956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:54.705526    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:10:57.183641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.183602    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:10:57.184131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.183856    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy" containerID="cri-o://86d9ac00ef450c769d9cf605ae4c76a40d9c629b83caacaee569f730eb085ead" gracePeriod=30
Oct 30 22:10:57.227821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.227763    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:10:57.228010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.227987    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy" containerID="cri-o://d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741" gracePeriod=30
Oct 30 22:10:57.231181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.231158    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 22:10:57.231360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.231339    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy" containerID="cri-o://98fd0325e12c46c7120404d25a8b619419fadeaa4c073aa2a9e81510bb4e268e" gracePeriod=30
Oct 30 22:10:57.273454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.273421    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:10:57.273620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.273601    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" containerID="cri-o://4c7d28fa4186979bcdf3247eeb56f5247087fd59eaca8aa1b60f0f1dd7a1824f" gracePeriod=30
Oct 30 22:10:57.374422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.374372    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 22:10:57.374641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.374613    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" containerID="cri-o://e06b01ffa3dce53a4620795d8dc3c493108cc98f993e531c54159259c360f3b2" gracePeriod=30
Oct 30 22:10:57.381280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.380858    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:10:57.381280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.381106    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" containerID="cri-o://5ec4029baa94f8b29c38d5b4768a62afb424ebf4b2925aead2f8fe6d36899294" gracePeriod=30
Oct 30 22:10:57.390596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.390558    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 22:10:57.391140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.391089    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" containerID="cri-o://eddff01c2da4c556f9e073e539c2c55f272284188f4d4573aa2310f2733df4b7" gracePeriod=30
Oct 30 22:10:57.395394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.395370    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 22:10:57.395597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.395567    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" containerID="cri-o://3af6224a9a688d5f3efc6e5afcce08bbe0c9ccffdd1b6f754a540f96424aa101" gracePeriod=30
Oct 30 22:10:57.798465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.798434    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:57.980591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:57.980549    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.323667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:58.323626    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:59.587126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:59.587085    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:59.587756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:59.587717    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:59.905856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:59.903019    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:10:59.926236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:10:59.925049    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:11:00.206245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:00.206154    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:11:01.780310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.779188    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:01.780310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.779275    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:01.830570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.830526    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:01.830802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.830583    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:01.892500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.892463    5510 patch_prober.go:29] interesting pod/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:01.892670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:01.892524    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:02.183465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.183375    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:11:02.184292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184272    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy"
Oct 30 22:11:02.184292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184293    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy"
Oct 30 22:11:02.184447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184336    5510 cpu_manager.go:414] "containerMap: removing container" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy"
Oct 30 22:11:02.184447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184341    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy"
Oct 30 22:11:02.184447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184360    5510 cpu_manager.go:414] "containerMap: removing container" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy"
Oct 30 22:11:02.184447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184366    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy"
Oct 30 22:11:02.184587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184460    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="61efb1f7-04b2-49bc-a633-de94b399353f" containerName="istio-proxy"
Oct 30 22:11:02.184587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184470    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93f619e-ee34-4897-a528-9a6971e23e24" containerName="istio-proxy"
Oct 30 22:11:02.184587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.184483    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b5f0a4-9839-4721-841c-0cc6f7139c54" containerName="istio-proxy"
Oct 30 22:11:02.188736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.188646    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.241207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.239563    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:02.286806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.286534    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:11:02.286806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.286724    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 22:11:02.292936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.292587    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:11:02.292936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.292714    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:02.292936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.292753    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-778d39d3-serving-cert\""
Oct 30 22:11:02.304487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.301959    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.304487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.302217    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.304487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.302251    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8ls\" (UniqueName: \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.304487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.302320    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.405052    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.405108    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8ls\" (UniqueName: \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.405186    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.405402    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.406392    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:02.406516    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-serving-cert: secret "maas-api-e2e-mt-778d39d3-serving-cert" not found
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:02.406590    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls podName:8249f19f-1397-40c4-8b2c-2a8d75d6ff36 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:02.906569665 +0000 UTC m=+1671.890476544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls") pod "maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" (UID: "8249f19f-1397-40c4-8b2c-2a8d75d6ff36") : secret "maas-api-e2e-mt-778d39d3-serving-cert" not found
Oct 30 22:11:02.433329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.427196    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:02.521527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.520382    5510 patch_prober.go:29] interesting pod/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 22:11:02.521527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.520448    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 22:11:02.527882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.527800    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8ls\" (UniqueName: \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.547891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.547755    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.623692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.623652    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:02.914692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.912156    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.931972    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932081    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932120    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932156    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932273    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932327    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932474    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932527    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932610    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\") pod \"02097807-fb39-45aa-8500-26b1afe0175c\" (UID: \"02097807-fb39-45aa-8500-26b1afe0175c\") "
Oct 30 22:11:02.933712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.932960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.936464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.936177    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.936464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.936249    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.936464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.936387    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:02.936464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.936441    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.936869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.936831    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.941422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.941272    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:02.941422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.941289    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:02.941422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.941389    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:02.942434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.942383    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:02.948114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:02.948065    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs" pod "02097807-fb39-45aa-8500-26b1afe0175c" (UID: "02097807-fb39-45aa-8500-26b1afe0175c"). InnerVolumeSpecName "kube-api-access-gdmjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.010171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.010051    5510 generic.go:356] "Generic (PLEG): container finished" podID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerID="98fd0325e12c46c7120404d25a8b619419fadeaa4c073aa2a9e81510bb4e268e" exitCode=0
Oct 30 22:11:03.010262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.010210    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerDied","Data":"98fd0325e12c46c7120404d25a8b619419fadeaa4c073aa2a9e81510bb4e268e"}
Oct 30 22:11:03.010298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.010287    5510 scope.go:122] "RemoveContainer" containerID="5955e135514f7510d295f6584d565c704e8ed4ea1f0f5bcec3e768b325e509ac"
Oct 30 22:11:03.014972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.014514    5510 generic.go:356] "Generic (PLEG): container finished" podID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerID="86d9ac00ef450c769d9cf605ae4c76a40d9c629b83caacaee569f730eb085ead" exitCode=0
Oct 30 22:11:03.014972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.014766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" event={"ID":"73350b1b-bc3e-436e-ba55-fcf13fc7dd21","Type":"ContainerDied","Data":"86d9ac00ef450c769d9cf605ae4c76a40d9c629b83caacaee569f730eb085ead"}
Oct 30 22:11:03.017136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.017011    5510 generic.go:356] "Generic (PLEG): container finished" podID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerID="4c7d28fa4186979bcdf3247eeb56f5247087fd59eaca8aa1b60f0f1dd7a1824f" exitCode=0
Oct 30 22:11:03.017136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.017103    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" event={"ID":"97c8a305-c1a8-4a0f-8d65-b707833c9d3c","Type":"ContainerDied","Data":"4c7d28fa4186979bcdf3247eeb56f5247087fd59eaca8aa1b60f0f1dd7a1824f"}
Oct 30 22:11:03.019051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.019026    5510 generic.go:356] "Generic (PLEG): container finished" podID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerID="eddff01c2da4c556f9e073e539c2c55f272284188f4d4573aa2310f2733df4b7" exitCode=0
Oct 30 22:11:03.019151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.019097    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" event={"ID":"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a","Type":"ContainerDied","Data":"eddff01c2da4c556f9e073e539c2c55f272284188f4d4573aa2310f2733df4b7"}
Oct 30 22:11:03.020922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.020896    5510 generic.go:356] "Generic (PLEG): container finished" podID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerID="5ec4029baa94f8b29c38d5b4768a62afb424ebf4b2925aead2f8fe6d36899294" exitCode=0
Oct 30 22:11:03.021027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.020980    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" event={"ID":"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba","Type":"ContainerDied","Data":"5ec4029baa94f8b29c38d5b4768a62afb424ebf4b2925aead2f8fe6d36899294"}
Oct 30 22:11:03.024125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.024002    5510 generic.go:356] "Generic (PLEG): container finished" podID="8b740417-1017-43a0-840f-57ca770df230" containerID="e06b01ffa3dce53a4620795d8dc3c493108cc98f993e531c54159259c360f3b2" exitCode=0
Oct 30 22:11:03.024125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.024070    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" event={"ID":"8b740417-1017-43a0-840f-57ca770df230","Type":"ContainerDied","Data":"e06b01ffa3dce53a4620795d8dc3c493108cc98f993e531c54159259c360f3b2"}
Oct 30 22:11:03.026409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.026206    5510 generic.go:356] "Generic (PLEG): container finished" podID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerID="3af6224a9a688d5f3efc6e5afcce08bbe0c9ccffdd1b6f754a540f96424aa101" exitCode=0
Oct 30 22:11:03.026409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.026345    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" event={"ID":"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7","Type":"ContainerDied","Data":"3af6224a9a688d5f3efc6e5afcce08bbe0c9ccffdd1b6f754a540f96424aa101"}
Oct 30 22:11:03.083699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.028441    5510 generic.go:356] "Generic (PLEG): container finished" podID="02097807-fb39-45aa-8500-26b1afe0175c" containerID="d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741" exitCode=0
Oct 30 22:11:03.083699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.028574    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" event={"ID":"02097807-fb39-45aa-8500-26b1afe0175c","Type":"ContainerDied","Data":"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"}
Oct 30 22:11:03.083699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.028615    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57" event={"ID":"02097807-fb39-45aa-8500-26b1afe0175c","Type":"ContainerDied","Data":"d38c590c4c9959915749940d4ecd81ca7c1075a2fb5c87442d3c91428cb2cc7a"}
Oct 30 22:11:03.083699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.028699    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"
Oct 30 22:11:03.113129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.113093    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116610    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116644    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097807-fb39-45aa-8500-26b1afe0175c-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116658    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116672    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116684    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116698    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116711    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097807-fb39-45aa-8500-26b1afe0175c-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116724    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097807-fb39-45aa-8500-26b1afe0175c-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.116740    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdmjs\" (UniqueName: \"kubernetes.io/projected/02097807-fb39-45aa-8500-26b1afe0175c-kube-api-access-gdmjs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.125819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.120745    5510 scope.go:122] "RemoveContainer" containerID="d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"
Oct 30 22:11:03.231810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.204486    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:11:03.234876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.225023    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.325945    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326024    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326074    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326131    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326216    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326253    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326288    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326319    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326464    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326510    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326547    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326612    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326645    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326693    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\") pod \"a7311c13-cd2c-43e4-8dca-17fb4be77766\" (UID: \"a7311c13-cd2c-43e4-8dca-17fb4be77766\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326751    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326836    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.327644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.326963    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\") pod \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\" (UID: \"73350b1b-bc3e-436e-ba55-fcf13fc7dd21\") "
Oct 30 22:11:03.328891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.327831    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.328891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.328036    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.328891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.328202    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.328891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.328382    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.331299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.331269    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.339554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.339513    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.339651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.339591    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.340595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.340464    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.340843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.340817    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.342949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.342465    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.342949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.342669    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.343114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.342977    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.343266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.343241    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.344809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.344373    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "kube-api-access-mtfnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.347349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.347313    5510 scope.go:122] "RemoveContainer" containerID="d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"
Oct 30 22:11:03.349882 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:03.349851    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741\": container with ID starting with d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741 not found: ID does not exist" containerID="d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"
Oct 30 22:11:03.349981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.349895    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741"} err="failed to get container status \"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741\": rpc error: code = NotFound desc = could not find container \"d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741\": container with ID starting with d425a84b18a073cb040d742c6d5753c8071c350e67a049fdedf9e8f11de54741 not found: ID does not exist"
Oct 30 22:11:03.382260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.366878    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.397610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.397558    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.400059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.399995    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token" pod "a7311c13-cd2c-43e4-8dca-17fb4be77766" (UID: "a7311c13-cd2c-43e4-8dca-17fb4be77766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.401547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.401433    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn" pod "73350b1b-bc3e-436e-ba55-fcf13fc7dd21" (UID: "73350b1b-bc3e-436e-ba55-fcf13fc7dd21"). InnerVolumeSpecName "kube-api-access-lg8xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435550    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435581    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435602    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435619    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435634    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435649    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfnh\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-kube-api-access-mtfnh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435664    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435680    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435694    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg8xn\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-kube-api-access-lg8xn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435708    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435723    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435737    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435750    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7311c13-cd2c-43e4-8dca-17fb4be77766-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435763    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7311c13-cd2c-43e4-8dca-17fb4be77766-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435793    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435807    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435823    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73350b1b-bc3e-436e-ba55-fcf13fc7dd21-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.436549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.435837    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7311c13-cd2c-43e4-8dca-17fb4be77766-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.504556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.504523    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 22:11:03.513670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.513628    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:11:03.516159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.516135    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:11:03.526955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.526871    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537427    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537490    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537558    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537589    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537642    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537689    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537723    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537759    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537839    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537900    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.537957    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538004    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538034    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538066    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538138    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538146    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538211    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538253    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538285    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538398    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\") pod \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\" (UID: \"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba\") "
Oct 30 22:11:03.538423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538437    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538574    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538661    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538747    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\") pod \"8b740417-1017-43a0-840f-57ca770df230\" (UID: \"8b740417-1017-43a0-840f-57ca770df230\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538794    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.538855    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\") pod \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\" (UID: \"97c8a305-c1a8-4a0f-8d65-b707833c9d3c\") "
Oct 30 22:11:03.539971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.539670    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.540949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.540675    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.541333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.541088    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542082    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542098    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542273    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542291    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542468    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.542708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.542559    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.545143 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.544432    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.545143 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.544679    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.546580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.546322    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.548430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.548217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.549039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.548641    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.550186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.550069    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.550463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.550283    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.550463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.550309    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.550699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.550655    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.551274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.551206    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "kube-api-access-d9jxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.551274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.551212    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "kube-api-access-jshpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.551274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.551256    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.551493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.551472    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.558043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.557941    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "kube-api-access-cqxc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.605862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.605820    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.606961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.606912    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy" pod "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" (UID: "ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.608275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.608236    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo" pod "97c8a305-c1a8-4a0f-8d65-b707833c9d3c" (UID: "97c8a305-c1a8-4a0f-8d65-b707833c9d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.608773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.608749    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy" pod "8b740417-1017-43a0-840f-57ca770df230" (UID: "8b740417-1017-43a0-840f-57ca770df230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.622873 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.622844    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 22:11:03.640987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.640953    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.640995    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641034    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641136    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641154    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641190    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641220    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641253    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641323    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641362    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641409    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641457    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641518    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641543    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\") pod \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\" (UID: \"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7\") "
Oct 30 22:11:03.641656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.641578    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\") pod \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\" (UID: \"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a\") "
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642005    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642019    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642028    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642037    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642045    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642054    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642062    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b740417-1017-43a0-840f-57ca770df230-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642071    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxc6\" (UniqueName: \"kubernetes.io/projected/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-kube-api-access-cqxc6\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642080    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642090    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshpk\" (UniqueName: \"kubernetes.io/projected/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-kube-api-access-jshpk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642098    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642106    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642114    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642123    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642132    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642140    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642147    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642156    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642164    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/8b740417-1017-43a0-840f-57ca770df230-kube-api-access-d9jxl\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642173    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642183    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642191    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c8a305-c1a8-4a0f-8d65-b707833c9d3c-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642199    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642208    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642217    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b740417-1017-43a0-840f-57ca770df230-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642225    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b740417-1017-43a0-840f-57ca770df230-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.642874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642830    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.643394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.642998    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.643394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.643184    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.643675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.643544    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.643903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.643874    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.644170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.644146    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.644369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.644347    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.644558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.644536    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.645312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.645286    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.645524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.645487    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.645761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.645732    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.646005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.645954    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.646005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.645952    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "kube-api-access-zrztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.646150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.646127    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "kube-api-access-xpfhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.646205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.646147    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.646269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.646249    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.647298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.647261    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy" pod "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" (UID: "6c8ed86c-a90b-4fce-9c19-e6de582c2c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:03.647298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.647281    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo" pod "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" (UID: "04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:03.744115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744079    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfhp\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-kube-api-access-xpfhp\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744114    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744126    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744139    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrztk\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-kube-api-access-zrztk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744153    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744165    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744191    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744206    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744219    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744231    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744243    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744254    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744266    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744277    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744289    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744303    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744319    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.744352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.744332    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.841359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.841320    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:11:03.900364 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:11:03.900329    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8249f19f_1397_40c4_8b2c_2a8d75d6ff36.slice/crio-61dfb39a6730aa84add3f84d4050d7602628e069ecb08e0bc9a3038485139f6d WatchSource:0}: Error finding container 61dfb39a6730aa84add3f84d4050d7602628e069ecb08e0bc9a3038485139f6d: Status 404 returned error can't find the container with id 61dfb39a6730aa84add3f84d4050d7602628e069ecb08e0bc9a3038485139f6d
Oct 30 22:11:03.902095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.902068    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:03.984900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.984862    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:11:03.990773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:03.990745    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-qbm57"]
Oct 30 22:11:04.037313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.037227    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"
Oct 30 22:11:04.037313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.037254    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj" event={"ID":"73350b1b-bc3e-436e-ba55-fcf13fc7dd21","Type":"ContainerDied","Data":"5ca618d4de19a7fd39647209d2709d533526b9f028fb1334533335ffe5a892fe"}
Oct 30 22:11:04.037313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.037314    5510 scope.go:122] "RemoveContainer" containerID="86d9ac00ef450c769d9cf605ae4c76a40d9c629b83caacaee569f730eb085ead"
Oct 30 22:11:04.038776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.038756    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"
Oct 30 22:11:04.038914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.038793    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49" event={"ID":"97c8a305-c1a8-4a0f-8d65-b707833c9d3c","Type":"ContainerDied","Data":"3909adbbf3d940bd7693680e0458ceae859d5e41f01fa3c6729388c913a0eff6"}
Oct 30 22:11:04.040544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.040518    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"
Oct 30 22:11:04.040655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.040541    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb" event={"ID":"6c8ed86c-a90b-4fce-9c19-e6de582c2c2a","Type":"ContainerDied","Data":"62f7872c9508374ab4a4b4f5343804d58362c59663dbef2775ad276757965285"}
Oct 30 22:11:04.042880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.042822    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp" event={"ID":"ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba","Type":"ContainerDied","Data":"1f3d495d847271332d33b3371fae1b1993095e4ba164e98695d70e3e2d0f19a1"}
Oct 30 22:11:04.043431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.043415    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"
Oct 30 22:11:04.080122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.044675    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2" event={"ID":"8b740417-1017-43a0-840f-57ca770df230","Type":"ContainerDied","Data":"94395091c7cb74b82ca6e3e2e6641cc72e4b0914727dab467b87879baa21da2e"}
Oct 30 22:11:04.080122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.044828    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"
Oct 30 22:11:04.080122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.065238    5510 scope.go:122] "RemoveContainer" containerID="4c7d28fa4186979bcdf3247eeb56f5247087fd59eaca8aa1b60f0f1dd7a1824f"
Oct 30 22:11:04.084446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.084396    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"
Oct 30 22:11:04.084573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.084455    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg" event={"ID":"04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7","Type":"ContainerDied","Data":"bb23b64a15c152de1b3f6fefb6dc21c57911051cf6cb56eb672d6a054709ebce"}
Oct 30 22:11:04.086351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.086306    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" event={"ID":"8249f19f-1397-40c4-8b2c-2a8d75d6ff36","Type":"ContainerStarted","Data":"61dfb39a6730aa84add3f84d4050d7602628e069ecb08e0bc9a3038485139f6d"}
Oct 30 22:11:04.101510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.101464    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq" event={"ID":"a7311c13-cd2c-43e4-8dca-17fb4be77766","Type":"ContainerDied","Data":"3981ee295a9e5da8bfb38a449c266e878f2f1dca3b5303b779f2d0fcb1e8ae4e"}
Oct 30 22:11:04.101692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.101672    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"
Oct 30 22:11:04.103754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.103524    5510 scope.go:122] "RemoveContainer" containerID="eddff01c2da4c556f9e073e539c2c55f272284188f4d4573aa2310f2733df4b7"
Oct 30 22:11:04.136316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.136215    5510 scope.go:122] "RemoveContainer" containerID="5ec4029baa94f8b29c38d5b4768a62afb424ebf4b2925aead2f8fe6d36899294"
Oct 30 22:11:04.146764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.146735    5510 scope.go:122] "RemoveContainer" containerID="e06b01ffa3dce53a4620795d8dc3c493108cc98f993e531c54159259c360f3b2"
Oct 30 22:11:04.187815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.187757    5510 scope.go:122] "RemoveContainer" containerID="3af6224a9a688d5f3efc6e5afcce08bbe0c9ccffdd1b6f754a540f96424aa101"
Oct 30 22:11:04.198716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.198696    5510 scope.go:122] "RemoveContainer" containerID="98fd0325e12c46c7120404d25a8b619419fadeaa4c073aa2a9e81510bb4e268e"
Oct 30 22:11:04.346879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.346793    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 22:11:04.401812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.401206    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-5hdt2"]
Oct 30 22:11:04.472706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.472606    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:11:04.487853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.478385    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-dw6tp"]
Oct 30 22:11:04.582145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.582110    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:11:04.585282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.585251    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-k87nj"]
Oct 30 22:11:04.747255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.747220    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 22:11:04.773819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.752315    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-jpwdg"]
Oct 30 22:11:04.881486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.881449    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:11:04.884213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.884189    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-vxw49"]
Oct 30 22:11:04.984110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.984068    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 22:11:04.987404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:04.987371    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5402dd-openshift-default-f6ff5b9d5-mkkwq"]
Oct 30 22:11:05.090412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.090322    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 22:11:05.094827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.094774    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d1f91-openshift-default-7967f7cd55-wh8mb"]
Oct 30 22:11:05.128509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.128480    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02097807-fb39-45aa-8500-26b1afe0175c" path="/var/lib/kubelet/pods/02097807-fb39-45aa-8500-26b1afe0175c/volumes"
Oct 30 22:11:05.129343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.129322    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" path="/var/lib/kubelet/pods/04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7/volumes"
Oct 30 22:11:05.130110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.130091    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" path="/var/lib/kubelet/pods/6c8ed86c-a90b-4fce-9c19-e6de582c2c2a/volumes"
Oct 30 22:11:05.131496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.131479    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" path="/var/lib/kubelet/pods/73350b1b-bc3e-436e-ba55-fcf13fc7dd21/volumes"
Oct 30 22:11:05.132184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.132165    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b740417-1017-43a0-840f-57ca770df230" path="/var/lib/kubelet/pods/8b740417-1017-43a0-840f-57ca770df230/volumes"
Oct 30 22:11:05.132924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.132907    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" path="/var/lib/kubelet/pods/97c8a305-c1a8-4a0f-8d65-b707833c9d3c/volumes"
Oct 30 22:11:05.134045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.134025    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" path="/var/lib/kubelet/pods/a7311c13-cd2c-43e4-8dca-17fb4be77766/volumes"
Oct 30 22:11:05.134748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:05.134729    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" path="/var/lib/kubelet/pods/ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba/volumes"
Oct 30 22:11:07.144418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:07.144327    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" event={"ID":"8249f19f-1397-40c4-8b2c-2a8d75d6ff36","Type":"ContainerStarted","Data":"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"}
Oct 30 22:11:07.144418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:07.144379    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:07.317974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:07.317939    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:11.174923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.174832    5510 generic.go:356] "Generic (PLEG): container finished" podID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerID="e85ce77043b8aff34969045c94cbc522c79b126471611190a18616ffe9fe6ed4" exitCode=0
Oct 30 22:11:11.174923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.174909    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerDied","Data":"e85ce77043b8aff34969045c94cbc522c79b126471611190a18616ffe9fe6ed4"}
Oct 30 22:11:11.175462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.175445    5510 scope.go:122] "RemoveContainer" containerID="e85ce77043b8aff34969045c94cbc522c79b126471611190a18616ffe9fe6ed4"
Oct 30 22:11:11.177109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.177081    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:11.177240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.177159    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:11.177240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.177207    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:11.378757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.378704    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" podStartSLOduration=6.429589853 podStartE2EDuration="9.378690137s" totalImagesPullingTime="2.949100284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:07.269113695 +0000 UTC m=+1676.253020573" watchObservedRunningTime="2026-10-30 22:11:11.378690137 +0000 UTC m=+1680.362597010"
Oct 30 22:11:11.808009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:11.807031    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:12.189949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:12.189865    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerStarted","Data":"e4ea0946269544fb73e3d6ba5b2a500b1858dcce7dd0fc224fb93ad0207509b4"}
Oct 30 22:11:12.417211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:12.417167    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:11:12.418059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:12.418034    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:11:12.418182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:12.418091    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:11:13.182991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:13.182950    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:11:13.314946 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:13.314871    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:11:13.415797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:13.415754    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:11:13.416048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:13.416026    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:11:14.415821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:14.415757    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:11:14.416219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:14.415850    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:11:14.509165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:14.509132    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:11:15.417033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:15.416996    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:11:15.417422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:15.417055    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:11:16.417187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:16.417149    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:11:16.417626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:16.417211    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:11:16.492863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:16.492699    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:16.698514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:16.698418    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:17.420012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:17.419986    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:11:17.421103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:17.421080    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:11:17.497450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:17.497419    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:17.866460 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:17.866430    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:18.071455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:18.071420    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:18.907959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:18.907923    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:21.564452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:21.564422    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:22.619747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.619715    5510 scope.go:122] "RemoveContainer" containerID="28ead4fdcd0c737479272e92d0f1a287b72815d7631cf2aeca80e6ba666ed55f"
Oct 30 22:11:22.629816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.629773    5510 scope.go:122] "RemoveContainer" containerID="c9d6a7eeac1ae41b46b5a72e5d35f6c1f833640825c9e154699cc58cc48e7785"
Oct 30 22:11:22.638661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.638639    5510 scope.go:122] "RemoveContainer" containerID="4ece35a76bb507b7986e5ebe3327c39fbf91cd58d17832e59c502469fb854316"
Oct 30 22:11:22.647441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.647418    5510 scope.go:122] "RemoveContainer" containerID="b9e8f4ff56302f69af693278748dedd08b71387f76607c59a3965c95c8af669c"
Oct 30 22:11:22.659850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.659816    5510 scope.go:122] "RemoveContainer" containerID="fc54d02be1ac9992f1fb326633bc76db2a8db40a34c4f8d74f488afc4aa22983"
Oct 30 22:11:22.672884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.672864    5510 scope.go:122] "RemoveContainer" containerID="7c17bf86c7dd3ea871814c18aaee4eefff58b46d96dff0d4e84c8490c38abf8f"
Oct 30 22:11:22.682584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:22.682561    5510 scope.go:122] "RemoveContainer" containerID="139fa99be4104efba77a84a9b0c2a29f98882353e5e49d175f205bb5233ad8f1"
Oct 30 22:11:25.775808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:25.772293    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:26.501470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:26.501436    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:26.779426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:26.779324    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:31.258238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:31.258156    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:31.430441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:31.430406    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:33.382422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.381681    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": read tcp 10.133.0.2:47508->10.133.0.155:15021: read: connection reset by peer" start-of-body=
Oct 30 22:11:33.382422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.381770    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": read tcp 10.133.0.2:47508->10.133.0.155:15021: read: connection reset by peer"
Oct 30 22:11:33.521023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.520959    5510 generic.go:356] "Generic (PLEG): container finished" podID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerID="929756a5e3b9cd713a388eac07a58f8f674f9175da24d0b581bcd994d83f021d" exitCode=0
Oct 30 22:11:33.521157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.521031    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerDied","Data":"929756a5e3b9cd713a388eac07a58f8f674f9175da24d0b581bcd994d83f021d"}
Oct 30 22:11:33.521446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.521433    5510 scope.go:122] "RemoveContainer" containerID="929756a5e3b9cd713a388eac07a58f8f674f9175da24d0b581bcd994d83f021d"
Oct 30 22:11:33.523067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.523038    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:33.523139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.523106    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:33.523181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:33.523139    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:34.530663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:34.530629    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerStarted","Data":"8fffcecba968014675e1846c842c1c8f785d0c9e14417026ab00de9c63113931"}
Oct 30 22:11:35.171803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:35.171223    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:11:35.172809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:35.172050    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:11:35.172809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:35.172115    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:11:36.073383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:36.073343    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:36.164815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:36.163133    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:11:36.164815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:36.163212    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:11:37.172204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:37.172161    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:11:37.172833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:37.172771    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:11:38.176961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:38.176923    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:11:38.177591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:38.177530    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:11:39.175055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:39.175018    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:11:39.176905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:39.176865    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:11:40.184080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.184049    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:11:40.185731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.185705    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:11:40.186814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.186766    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:11:40.487017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.486981    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:11:40.487812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487766    5510 cpu_manager.go:414] "containerMap: removing container" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy"
Oct 30 22:11:40.487812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487805    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487821    5510 cpu_manager.go:414] "containerMap: removing container" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487827    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487866    5510 cpu_manager.go:414] "containerMap: removing container" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487873    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487884    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487888    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487895    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487900    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487912    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487917    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487926    5510 cpu_manager.go:414] "containerMap: removing container" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487931    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487940    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487945    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487951    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy"
Oct 30 22:11:40.487989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.487957    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488072    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488084    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8ed86c-a90b-4fce-9c19-e6de582c2c2a" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488091    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="73350b1b-bc3e-436e-ba55-fcf13fc7dd21" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488097    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b740417-1017-43a0-840f-57ca770df230" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488102    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3b11ae-eab3-4f2a-ad97-141b5f16e7ba" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488114    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b23a6f-a9dc-4ac5-a7ba-159ea4ddd7e7" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488121    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c8a305-c1a8-4a0f-8d65-b707833c9d3c" containerName="istio-proxy"
Oct 30 22:11:40.488540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.488133    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="02097807-fb39-45aa-8500-26b1afe0175c" containerName="istio-proxy"
Oct 30 22:11:40.491998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.491977    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.543544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.543516    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-02bd3b-serving-cert\""
Oct 30 22:11:40.571456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.571420    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:11:40.636331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.636297    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrkt\" (UniqueName: \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.636559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.636534    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.636648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.636578    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.636734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.636708    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.717181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.717143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.741076    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.741136    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.741245    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.741427    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrkt\" (UniqueName: \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:40.742014    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-serving-cert: secret "maas-api-e2e-shared-b-w5-02bd3b-serving-cert" not found
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:40.742081    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls podName:187ccd21-bb05-49cd-9f6a-8f3b2795b1e1 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:41.242060339 +0000 UTC m=+1710.225967207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls") pod "maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" (UID: "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1") : secret "maas-api-e2e-shared-b-w5-02bd3b-serving-cert" not found
Oct 30 22:11:40.748856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.743343    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.796030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.796001    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.820536 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.820500    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrkt\" (UniqueName: \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:40.893182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:40.893016    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:41.112726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:41.112647    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:41.248112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:41.248081    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:41.250448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:41.250422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:41.407118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:41.407048    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:41.804024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:41.803984    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:11:41.811722 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:11:41.811688    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187ccd21_bb05_49cd_9f6a_8f3b2795b1e1.slice/crio-86d7d3cc0fb3b515031fa10775800c4d47a13cd6bde8b9950f1b015e99fe64d9 WatchSource:0}: Error finding container 86d7d3cc0fb3b515031fa10775800c4d47a13cd6bde8b9950f1b015e99fe64d9: Status 404 returned error can't find the container with id 86d7d3cc0fb3b515031fa10775800c4d47a13cd6bde8b9950f1b015e99fe64d9
Oct 30 22:11:42.610832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:42.610766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" event={"ID":"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1","Type":"ContainerStarted","Data":"86d7d3cc0fb3b515031fa10775800c4d47a13cd6bde8b9950f1b015e99fe64d9"}
Oct 30 22:11:43.287756 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:43.287720    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:11:43.626859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:43.626738    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" event={"ID":"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1","Type":"ContainerStarted","Data":"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"}
Oct 30 22:11:43.627580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:43.627560    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:43.744421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:43.744364    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" podStartSLOduration=2.867245107 podStartE2EDuration="3.744345505s" totalImagesPullingTime="877.100398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:43.741541902 +0000 UTC m=+1712.725448779" watchObservedRunningTime="2026-10-30 22:11:43.744345505 +0000 UTC m=+1712.728252382"
Oct 30 22:11:45.489143 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.489111    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:11:45.490478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.490453    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7311c13-cd2c-43e4-8dca-17fb4be77766" containerName="istio-proxy"
Oct 30 22:11:45.494969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.494950    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.590993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.590948    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-6ea8da-serving-cert\""
Oct 30 22:11:45.595341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.595304    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:11:45.618511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.618476    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.618672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.618582    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.618775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.618756    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82ff\" (UniqueName: \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.619020 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.618992    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.696011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.695971    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:45.722519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.722480    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82ff\" (UniqueName: \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.722732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.722626    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.722732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.722690    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.722921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.722766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.724150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.724113    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.724272 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:45.724207    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-serving-cert: secret "maas-api-e2e-shared-a-w6-6ea8da-serving-cert" not found
Oct 30 22:11:45.724272 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:11:45.724264    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls podName:118964a8-172f-4d71-a507-addbaca906ef nodeName:}" failed. No retries permitted until 2026-10-30 22:11:46.224246802 +0000 UTC m=+1715.208153656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls") pod "maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" (UID: "118964a8-172f-4d71-a507-addbaca906ef") : secret "maas-api-e2e-shared-a-w6-6ea8da-serving-cert" not found
Oct 30 22:11:45.742366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.742278    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.777205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.777163    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82ff\" (UniqueName: \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:45.959408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:45.958703    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:46.129772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:46.129686    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:46.243818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:46.243753    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:46.246219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:46.246196    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:46.412059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:46.411979    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:46.790603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:46.790564    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:11:46.796985 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:11:46.796957    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118964a8_172f_4d71_a507_addbaca906ef.slice/crio-30572e707e89f84362e21795ecdafcc03d9fb8ebd160bae54f8f700ba16a5b92 WatchSource:0}: Error finding container 30572e707e89f84362e21795ecdafcc03d9fb8ebd160bae54f8f700ba16a5b92: Status 404 returned error can't find the container with id 30572e707e89f84362e21795ecdafcc03d9fb8ebd160bae54f8f700ba16a5b92
Oct 30 22:11:47.658256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:47.658214    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" event={"ID":"118964a8-172f-4d71-a507-addbaca906ef","Type":"ContainerStarted","Data":"30572e707e89f84362e21795ecdafcc03d9fb8ebd160bae54f8f700ba16a5b92"}
Oct 30 22:11:48.673830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:48.670479    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" event={"ID":"118964a8-172f-4d71-a507-addbaca906ef","Type":"ContainerStarted","Data":"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"}
Oct 30 22:11:48.673830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:48.671798    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:48.789003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:48.788952    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" podStartSLOduration=2.8668944 podStartE2EDuration="3.788937338s" totalImagesPullingTime="922.042938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:48.786448596 +0000 UTC m=+1717.770355473" watchObservedRunningTime="2026-10-30 22:11:48.788937338 +0000 UTC m=+1717.772844224"
Oct 30 22:11:49.639687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:49.639655    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:11:51.145543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:51.145466    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:53.219054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:53.218964    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:53.692098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:53.692016    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.418543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.418300    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:11:54.452855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.452823    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.518345 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.518309    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:11:54.574804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.566035    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e66edb-openshift-default-dockercfg-nwr4z\""
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.583657    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzp54\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.583826    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.583900    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.583966    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.583996    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.584073    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.584127    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.584154    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.588822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.584200    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.613217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.596626    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:54.685945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.685646    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzp54\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.686443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.686295    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.686761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.686699    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.687074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.687024    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.687531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.687492    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.690042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.689744    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.690042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.689843    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.690042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.689876    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.690538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.690457    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.692012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.691990    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.692611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.692570    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.693051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.693014    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.697492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.693393    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.698258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.698235    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.700524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.700438    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.701508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.701484    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.756856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.754589    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.756856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.755560    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzp54\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\") pod \"e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") " pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.780284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.780245    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:54.796600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:54.796556    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:55.219862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.219832    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:11:55.225682 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:11:55.225657    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2a7f65_1d47_4584_85b9_238d9a98bedb.slice/crio-9cfaab8a2c47d24e9b7740da43a3043789ef98838ec02742aac8a09cdd5de7b2 WatchSource:0}: Error finding container 9cfaab8a2c47d24e9b7740da43a3043789ef98838ec02742aac8a09cdd5de7b2: Status 404 returned error can't find the container with id 9cfaab8a2c47d24e9b7740da43a3043789ef98838ec02742aac8a09cdd5de7b2
Oct 30 22:11:55.228295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.228263    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:55.228378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.228324    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:55.228378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.228357    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:11:55.691747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.691653    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:11:55.754987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.754950    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerStarted","Data":"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"}
Oct 30 22:11:55.755148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.755012    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerStarted","Data":"9cfaab8a2c47d24e9b7740da43a3043789ef98838ec02742aac8a09cdd5de7b2"}
Oct 30 22:11:55.781213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.781182    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:55.936424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:55.936374    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podStartSLOduration=1.936360191 podStartE2EDuration="1.936360191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:55.932966379 +0000 UTC m=+1724.916873254" watchObservedRunningTime="2026-10-30 22:11:55.936360191 +0000 UTC m=+1724.920267064"
Oct 30 22:11:56.255907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:56.255855    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:56.787628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:56.787598    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:57.772216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:57.772171    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:57.773304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:57.773282    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:11:58.932476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:58.932401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:59.172529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:59.172492    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:59.393403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:11:59.393369    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:01.775642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:01.775608    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:01.999293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:01.999242    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:12:02.394858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.389558    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.616804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.612770    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.752813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.752203    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:12:02.831231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.831199    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:02.839147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.839089    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:12:02.876831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.876794    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:12:02.925353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.925274    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e66edb\""
Oct 30 22:12:02.962803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.942143    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:02.962803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.962045    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5dj\" (UniqueName: \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:02.962803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:02.962148    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.148834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.148361    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e66edb-dockercfg-nqppv\""
Oct 30 22:12:03.203805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.180563    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5dj\" (UniqueName: \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.203805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.180647    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.203805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.180705    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.203805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.180804    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9t44\" (UniqueName: \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.203805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.182088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.214805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.208736    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:12:03.302821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.294478    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.302821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.294602    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9t44\" (UniqueName: \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.302821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.300005    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.308505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.308469    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5dj\" (UniqueName: \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\") pod \"payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.382858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.379341    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9t44\" (UniqueName: \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\") pod \"payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.480808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.465026    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:03.515288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.515266    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:03.951701 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:03.951623    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:12:04.007430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.007390    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:12:04.013064 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:04.013042    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93e1346_4d17_4088_a623_fed64392a217.slice/crio-645bee064548bc0bf3fa07d3d2e2321575a72d70fcee60811b7e597bb5494bbb WatchSource:0}: Error finding container 645bee064548bc0bf3fa07d3d2e2321575a72d70fcee60811b7e597bb5494bbb: Status 404 returned error can't find the container with id 645bee064548bc0bf3fa07d3d2e2321575a72d70fcee60811b7e597bb5494bbb
Oct 30 22:12:04.915023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.914981    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" event={"ID":"eea08086-7b2c-4154-bee9-fc576bb6315d","Type":"ContainerStarted","Data":"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"}
Oct 30 22:12:04.915198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.915047    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" event={"ID":"eea08086-7b2c-4154-bee9-fc576bb6315d","Type":"ContainerStarted","Data":"8dc3971b96dad66c3bf1606b37e32442519a11e8bd346ac22f8f5096ffb598c6"}
Oct 30 22:12:04.915250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.915228    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:04.916770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.916743    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" event={"ID":"b93e1346-4d17-4088-a623-fed64392a217","Type":"ContainerStarted","Data":"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"}
Oct 30 22:12:04.916915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.916820    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:04.916915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:04.916843    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" event={"ID":"b93e1346-4d17-4088-a623-fed64392a217","Type":"ContainerStarted","Data":"645bee064548bc0bf3fa07d3d2e2321575a72d70fcee60811b7e597bb5494bbb"}
Oct 30 22:12:05.038971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:05.038914    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" podStartSLOduration=3.03890023 podStartE2EDuration="3.03890023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:05.035004905 +0000 UTC m=+1734.018911780" watchObservedRunningTime="2026-10-30 22:12:05.03890023 +0000 UTC m=+1734.022807106"
Oct 30 22:12:05.220806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:05.214685    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" podStartSLOduration=3.214662983 podStartE2EDuration="3.214662983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:05.211439521 +0000 UTC m=+1734.195346398" watchObservedRunningTime="2026-10-30 22:12:05.214662983 +0000 UTC m=+1734.198569860"
Oct 30 22:12:06.846916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:06.846884    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.095805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:07.094757    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"]
Oct 30 22:12:07.130754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:07.130344    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284b6f16-mclsp"]
Oct 30 22:12:09.141809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:09.141706    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01bf4dd-4de3-460f-ba49-6efffd96964d" path="/var/lib/kubelet/pods/a01bf4dd-4de3-460f-ba49-6efffd96964d/volumes"
Oct 30 22:12:09.901346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:09.901305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:10.597464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.597412    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:12:10.768805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.767749    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:10.865957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.842009    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.842195    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.870077    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871056    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871128    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871181    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871335    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25lf\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871376    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871523    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871563    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.871732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.871629    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.924598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.924566    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5fef81-openshift-default-dockercfg-kjd5w\""
Oct 30 22:12:10.974003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.973964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974010    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974120    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974170    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25lf\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974209    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974283    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974321    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974374    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974421    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974711    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.974985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.974965    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.975543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.975506    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.976091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.976066    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.977961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.977934    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:10.978270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:10.978247    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:11.033888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.033848    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:11.035576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.035549    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25lf\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\") pod \"e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") " pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:11.190854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.189875    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:11.245534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.245492    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5fef81-openshift-default-dockercfg-kjd5w\""
Oct 30 22:12:11.247614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.247590    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:11.576971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.576926    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:12:11.577175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.577028    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:11.737033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.736998    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:12:11.779151 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:11.779117    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8303af_8bf0_4abd_98e5_38b4384a4110.slice/crio-06bf7063d2b336ae3afcc9bc5c2a6b49bb879a93eb2173d2a6a4c7b79fa3516c WatchSource:0}: Error finding container 06bf7063d2b336ae3afcc9bc5c2a6b49bb879a93eb2173d2a6a4c7b79fa3516c: Status 404 returned error can't find the container with id 06bf7063d2b336ae3afcc9bc5c2a6b49bb879a93eb2173d2a6a4c7b79fa3516c
Oct 30 22:12:11.782908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.782864    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:11.783032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.782953    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:11.783032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.782995    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:11.939868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:11.939619    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:12.022954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.022912    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerStarted","Data":"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"}
Oct 30 22:12:12.023153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.022984    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerStarted","Data":"06bf7063d2b336ae3afcc9bc5c2a6b49bb879a93eb2173d2a6a4c7b79fa3516c"}
Oct 30 22:12:12.154874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.154749    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podStartSLOduration=2.154730845 podStartE2EDuration="2.154730845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:12.153521347 +0000 UTC m=+1741.137428224" watchObservedRunningTime="2026-10-30 22:12:12.154730845 +0000 UTC m=+1741.138637722"
Oct 30 22:12:12.248246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.248207    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:12.250030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.249999    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:12:12.250178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:12.250048    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:12:13.032856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.032819    5510 generic.go:356] "Generic (PLEG): container finished" podID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerID="956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e" exitCode=0
Oct 30 22:12:13.033278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.032869    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerDied","Data":"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"}
Oct 30 22:12:13.033396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.033379    5510 scope.go:122] "RemoveContainer" containerID="956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"
Oct 30 22:12:13.035038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.035005    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:13.035159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.035092    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:13.035159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.035140    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:13.256800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:13.254547    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:13.316215 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:12:13.316178    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:12:14.041650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.041618    5510 generic.go:356] "Generic (PLEG): container finished" podID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerID="dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50" exitCode=0
Oct 30 22:12:14.042125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.041689    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerDied","Data":"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"}
Oct 30 22:12:14.042125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.042118    5510 scope.go:122] "RemoveContainer" containerID="dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"
Oct 30 22:12:14.043941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.043912    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:14.044061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.043978    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:14.044061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.044011    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:14.044061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.044011    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerStarted","Data":"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"}
Oct 30 22:12:14.044238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.044223    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:14.045189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.045167    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:12:14.347097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.347008    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:12:14.347271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.347212    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:14.347340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:14.347268    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:15.079807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.061109    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerStarted","Data":"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"}
Oct 30 22:12:15.347168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.346222    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:15.347168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.346279    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:15.383725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.383646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.588827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.586504    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:15.684635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.683486    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:12:15.684635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.683587    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:15.684635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.683644    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:15.926115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.926080    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:12:15.927504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:15.927485    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:12:16.346495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:16.346465    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:16.346898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:16.346531    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:16.680863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:16.680353    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:16.680863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:16.680417    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:17.346354 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:17.346314    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:17.346534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:17.346381    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:17.453902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:17.423549    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:17.673860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:17.673087    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:17.673860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:17.673158    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:18.346284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:18.346251    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:18.346459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:18.346304    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:18.675263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:18.675167    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:18.676540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:18.676507    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:19.345819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:19.345762    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:12:19.346019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:19.345832    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 22:12:19.676858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:19.676115    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:19.676858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:19.676181    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:20.350045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:20.350013    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:12:20.350415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:20.350389    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:12:20.351082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:20.351061    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:12:20.674263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:20.674174    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 22:12:20.674978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:20.674941    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 22:12:21.753635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:21.752880    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:12:21.753635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:21.753527    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:12:21.757086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:21.755759    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:12:22.443028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:22.442993    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:22.690864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:22.690831    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.073838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.073800    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.251768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.251735    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:23.479452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.479416    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:12:23.486027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.485995    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.582249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.582205    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5fef81\""
Oct 30 22:12:23.638216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.638180    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5fef81-dockercfg-2xvzv\""
Oct 30 22:12:23.684796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.684732    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5w8x\" (UniqueName: \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.685307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.685230    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.688401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.688353    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:12:23.697609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.697578    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:12:23.711249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.711219    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.714832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.714804    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:12:23.787190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.787158    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.787369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.787229    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5w8x\" (UniqueName: \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.787420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.787369    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.787579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.787551    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmvt\" (UniqueName: \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.788085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.788068    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.841277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.841240    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5w8x\" (UniqueName: \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\") pod \"payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:23.888994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.888958    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmvt\" (UniqueName: \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.889159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.889108    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.889773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.889751    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:23.945443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:23.945412    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmvt\" (UniqueName: \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\") pod \"payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") " pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:24.047292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:24.047209    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:24.102884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:24.102847    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:24.442984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:24.442907    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:12:24.448565 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:24.448537    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c35fd26_9a8b_4dc8_85a1_e8889ca6ec9c.slice/crio-5da483bd9d55e2ef87332b7c5951a668cec1761829934e801333b2301abd3d0a WatchSource:0}: Error finding container 5da483bd9d55e2ef87332b7c5951a668cec1761829934e801333b2301abd3d0a: Status 404 returned error can't find the container with id 5da483bd9d55e2ef87332b7c5951a668cec1761829934e801333b2301abd3d0a
Oct 30 22:12:24.498118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:24.498076    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:12:24.512154 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:24.512122    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1297551d_b906_442e_91a7_2d41dff03574.slice/crio-be6d5e00312f031af9c3108893a5be74221f8da2236253552bc3e2ff897fbf9e WatchSource:0}: Error finding container be6d5e00312f031af9c3108893a5be74221f8da2236253552bc3e2ff897fbf9e: Status 404 returned error can't find the container with id be6d5e00312f031af9c3108893a5be74221f8da2236253552bc3e2ff897fbf9e
Oct 30 22:12:25.258956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.258922    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" event={"ID":"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c","Type":"ContainerStarted","Data":"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"}
Oct 30 22:12:25.259442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.258980    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" event={"ID":"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c","Type":"ContainerStarted","Data":"5da483bd9d55e2ef87332b7c5951a668cec1761829934e801333b2301abd3d0a"}
Oct 30 22:12:25.259442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.259010    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:25.260824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.260767    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" event={"ID":"1297551d-b906-442e-91a7-2d41dff03574","Type":"ContainerStarted","Data":"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"}
Oct 30 22:12:25.260933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.260851    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" event={"ID":"1297551d-b906-442e-91a7-2d41dff03574","Type":"ContainerStarted","Data":"be6d5e00312f031af9c3108893a5be74221f8da2236253552bc3e2ff897fbf9e"}
Oct 30 22:12:25.260933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.260885    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:25.370154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.370095    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" podStartSLOduration=2.370080964 podStartE2EDuration="2.370080964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:25.366912041 +0000 UTC m=+1754.350818917" watchObservedRunningTime="2026-10-30 22:12:25.370080964 +0000 UTC m=+1754.353987834"
Oct 30 22:12:25.475516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:25.475458    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" podStartSLOduration=2.475443272 podStartE2EDuration="2.475443272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:25.474443845 +0000 UTC m=+1754.458350711" watchObservedRunningTime="2026-10-30 22:12:25.475443272 +0000 UTC m=+1754.459350567"
Oct 30 22:12:26.891473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:26.891435    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:27.318859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:27.318810    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.150463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:28.150428    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:29.206082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:29.206050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.870938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:31.870873    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.931807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:31.931744    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:12:31.949046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:31.949015    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:31.998810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:31.998613    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:12:32.034762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.034731    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.034931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.034889    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.035015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.034985    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.035145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.035058    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhn5\" (UniqueName: \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.035145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.035098    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.035323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.035297    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.038340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.038317    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.039076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.039057    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-778d39d3\"/\"kube-root-ca.crt\""
Oct 30 22:12:32.039076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.039057    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-778d39d3\"/\"openshift-service-ca.crt\""
Oct 30 22:12:32.039198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.039086    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-778d39d3\"/\"e2e-lifecycle-model-778d39d3-kserve-self-signed-certs\""
Oct 30 22:12:32.046062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.046042    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-778d39d3\"/\"default-dockercfg-5rbkk\""
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.143692    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.143796    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.143848    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhn5\" (UniqueName: \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.143891    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.144085    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.146549    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.146600    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.146927    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.156859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.147139    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.162269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.162238    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.164351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.162746    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.206833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.205510    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhn5\" (UniqueName: \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\") pod \"e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") " pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.286744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.286702    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:32.675559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:32.675521    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:12:32.682215 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:32.682183    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41a4ee57_fafc_4e90_bfba_2e1811ae0f5e.slice/crio-f8cc5f6f00728c1dab0a2288f25d871f2125796267aebb94d5048beb59cd5352 WatchSource:0}: Error finding container f8cc5f6f00728c1dab0a2288f25d871f2125796267aebb94d5048beb59cd5352: Status 404 returned error can't find the container with id f8cc5f6f00728c1dab0a2288f25d871f2125796267aebb94d5048beb59cd5352
Oct 30 22:12:33.325501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.325464    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" event={"ID":"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e","Type":"ContainerStarted","Data":"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"}
Oct 30 22:12:33.326019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.325509    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" event={"ID":"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e","Type":"ContainerStarted","Data":"f8cc5f6f00728c1dab0a2288f25d871f2125796267aebb94d5048beb59cd5352"}
Oct 30 22:12:33.435955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.435895    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" podStartSLOduration=2.435882048 podStartE2EDuration="2.435882048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:33.433207471 +0000 UTC m=+1762.417114346" watchObservedRunningTime="2026-10-30 22:12:33.435882048 +0000 UTC m=+1762.419788923"
Oct 30 22:12:33.703871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.703763    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:12:33.708423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.708400    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790560    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790620    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790701    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790763    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790846    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.790939    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4d6\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.791001    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.817166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.791042    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929365    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929433    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929487    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4d6\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929548    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929588    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929669    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929703    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929849    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.929921    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.935801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.930416    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:33.940291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:33.939510    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.038161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.038121    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.047380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.047283    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-411ff1-openshift-default-dockercfg-kpfph\""
Oct 30 22:12:34.048215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.048185    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.048517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.048495    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.063420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.062186    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.079473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.079429    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:12:34.082722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.082694    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.087179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.087080    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.087611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.087411    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.102220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.102191    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4d6\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\") pod \"e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") " pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.290858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.289539    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:34.327827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.326306    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:34.360631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.360605    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.665637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.665557    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:34.771371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.771337    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:12:34.777505 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:34.777474    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be2cac5_c8d2_4d14_b775_ef3f9ed372f7.slice/crio-02f285d9a878cce92af0c1136eb1c7daf8fe63fac2b2c10e98d37df74b10d9d9 WatchSource:0}: Error finding container 02f285d9a878cce92af0c1136eb1c7daf8fe63fac2b2c10e98d37df74b10d9d9: Status 404 returned error can't find the container with id 02f285d9a878cce92af0c1136eb1c7daf8fe63fac2b2c10e98d37df74b10d9d9
Oct 30 22:12:34.780374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.780340    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:34.780473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.780408    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:34.780473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:34.780434    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:12:35.356853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:35.356816    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerStarted","Data":"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"}
Oct 30 22:12:35.357286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:35.356885    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerStarted","Data":"02f285d9a878cce92af0c1136eb1c7daf8fe63fac2b2c10e98d37df74b10d9d9"}
Oct 30 22:12:35.487129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:35.487071    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podStartSLOduration=2.487053694 podStartE2EDuration="2.487053694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:35.484408175 +0000 UTC m=+1764.468315052" watchObservedRunningTime="2026-10-30 22:12:35.487053694 +0000 UTC m=+1764.470960574"
Oct 30 22:12:36.271563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.271529    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:12:36.271756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.271669    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:12:36.327925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.327897    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:36.333633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.333610    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:36.336637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.336606    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.432277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.432169    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:36.437233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:36.435032    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:12:38.354262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:38.354228    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.786503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:38.786448    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:38.973548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:38.973510    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.167383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.167341    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.413179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.413143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.628021 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.604503    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.807806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.806919    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:12:41.817584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.817526    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:41.838727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.838695    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:41.838950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.838731    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl64n\" (UniqueName: \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:41.864061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.864028    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:12:41.866403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.866368    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-411ff1\""
Oct 30 22:12:42.002859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.985347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:42.003190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.003162    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl64n\" (UniqueName: \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:42.016805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.995058    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:12:42.028720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:41.988678    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:42.078196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.078159    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.113800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.110231    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:12:42.145745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.143799    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-411ff1-dockercfg-wvr6g\""
Oct 30 22:12:42.161835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.158717    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl64n\" (UniqueName: \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\") pod \"payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:42.219878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.213551    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57f22\" (UniqueName: \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.219878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.213614    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.286946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.286865    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:42.300303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.300275    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:42.315171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.315140    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.315390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.315375    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57f22\" (UniqueName: \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.315933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.315913    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.369989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.369946    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57f22\" (UniqueName: \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\") pod \"payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.437621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.437589    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:42.497796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.495849    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:12:42.515585 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.515552    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:42.833884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.833805    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:12:42.840089 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:42.840055    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5d6e3f_57b0_49ac_bbaf_97a609b78a04.slice/crio-994f57712c96828606be345e7e5e998a4f35ef0596ad6d854556f2895a557d25 WatchSource:0}: Error finding container 994f57712c96828606be345e7e5e998a4f35ef0596ad6d854556f2895a557d25: Status 404 returned error can't find the container with id 994f57712c96828606be345e7e5e998a4f35ef0596ad6d854556f2895a557d25
Oct 30 22:12:42.944352 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:42.944317    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:12:42.949919 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:12:42.949889    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef8a1d9_da05_4696_b7fb_15588f0c3efc.slice/crio-41e8e50972425b4c2320bf4f85d75aa08085b4512cc99aa4056ce143abeaf77e WatchSource:0}: Error finding container 41e8e50972425b4c2320bf4f85d75aa08085b4512cc99aa4056ce143abeaf77e: Status 404 returned error can't find the container with id 41e8e50972425b4c2320bf4f85d75aa08085b4512cc99aa4056ce143abeaf77e
Oct 30 22:12:43.282711 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:12:43.282677    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:12:43.488846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.488801    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" event={"ID":"fef8a1d9-da05-4696-b7fb-15588f0c3efc","Type":"ContainerStarted","Data":"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"}
Oct 30 22:12:43.489054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.488864    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" event={"ID":"fef8a1d9-da05-4696-b7fb-15588f0c3efc","Type":"ContainerStarted","Data":"41e8e50972425b4c2320bf4f85d75aa08085b4512cc99aa4056ce143abeaf77e"}
Oct 30 22:12:43.489054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.488949    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:43.491222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.491191    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" event={"ID":"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04","Type":"ContainerStarted","Data":"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"}
Oct 30 22:12:43.491351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.491249    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" event={"ID":"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04","Type":"ContainerStarted","Data":"994f57712c96828606be345e7e5e998a4f35ef0596ad6d854556f2895a557d25"}
Oct 30 22:12:43.604045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.603934    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" podStartSLOduration=2.603919535 podStartE2EDuration="2.603919535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:43.601544862 +0000 UTC m=+1772.585451739" watchObservedRunningTime="2026-10-30 22:12:43.603919535 +0000 UTC m=+1772.587826410"
Oct 30 22:12:43.736852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:43.728720    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" podStartSLOduration=2.728700749 podStartE2EDuration="2.728700749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:43.725368477 +0000 UTC m=+1772.709275356" watchObservedRunningTime="2026-10-30 22:12:43.728700749 +0000 UTC m=+1772.712607626"
Oct 30 22:12:44.250891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:44.250853    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:44.412678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:44.412646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:44.500432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:44.500398    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:45.840484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:45.840450    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:48.356475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:48.356437    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:12:50.701820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:50.696713    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:54.503986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:54.503954    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:12:55.514928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:55.514900    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:12:55.644803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:12:55.644754    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:00.456421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:00.456361    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:02.509680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:02.509636    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:13:02.898946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:02.898857    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:03.073597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:03.073549    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:05.335494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:05.335463    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:05.906378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:05.906340    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:06.847145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:06.847104    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:07.320713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.320672    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:07.510180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.510135    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" podUID="966ea468-60a5-4c66-8243-a008d56cca2a" containerName="payload-processing" containerID="cri-o://15caab9d76829ddd2ee7c36a9a2a23d7b19967e782cd2363db8bd770719cc7e5" gracePeriod=25
Oct 30 22:13:07.781950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.777339    5510 generic.go:356] "Generic (PLEG): container finished" podID="966ea468-60a5-4c66-8243-a008d56cca2a" containerID="15caab9d76829ddd2ee7c36a9a2a23d7b19967e782cd2363db8bd770719cc7e5" exitCode=0
Oct 30 22:13:07.781950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.777503    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" event={"ID":"966ea468-60a5-4c66-8243-a008d56cca2a","Type":"ContainerDied","Data":"15caab9d76829ddd2ee7c36a9a2a23d7b19967e782cd2363db8bd770719cc7e5"}
Oct 30 22:13:07.860562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.860534    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:13:07.930065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.929971    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:13:07.947057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:07.931475    5510 cpu_manager.go:414] "containerMap: removing container" podUID="966ea468-60a5-4c66-8243-a008d56cca2a" containerName="payload-processing"
Oct 30 22:13:08.226633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.226604    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966ea468-60a5-4c66-8243-a008d56cca2a" containerName="payload-processing"
Oct 30 22:13:08.228091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.023881    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\" (UniqueName: \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\") pod \"966ea468-60a5-4c66-8243-a008d56cca2a\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") "
Oct 30 22:13:08.228091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.227263    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="966ea468-60a5-4c66-8243-a008d56cca2a" containerName="payload-processing"
Oct 30 22:13:08.228091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.227366    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\") pod \"966ea468-60a5-4c66-8243-a008d56cca2a\" (UID: \"966ea468-60a5-4c66-8243-a008d56cca2a\") "
Oct 30 22:13:08.228509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.148291    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.228651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.063023    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg" pod "966ea468-60a5-4c66-8243-a008d56cca2a" (UID: "966ea468-60a5-4c66-8243-a008d56cca2a"). InnerVolumeSpecName "kube-api-access-bwsxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:08.228651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.228411    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume" pod "966ea468-60a5-4c66-8243-a008d56cca2a" (UID: "966ea468-60a5-4c66-8243-a008d56cca2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:08.232652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.232579    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:13:08.233919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.233892    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.288185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.287964    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-c9dc10-openshift-default-dockercfg-6z2zz\""
Oct 30 22:13:08.328845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.328767    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.328918    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gk5\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.328991    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329042    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329183    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329290    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329429    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329470    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329487    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.329840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329812    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwsxg\" (UniqueName: \"kubernetes.io/projected/966ea468-60a5-4c66-8243-a008d56cca2a-kube-api-access-bwsxg\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.329954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.329845    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ea468-60a5-4c66-8243-a008d56cca2a-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:08.378725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.378691    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.430996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.430964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431025    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gk5\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431069    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431087    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431160    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431228    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431340    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431378    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431408    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431485    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431741    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.431900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.431819    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.432106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.432079    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.432230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.432182    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.433948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.433928    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.434071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.433992    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.484459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.484391    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.484612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.484575    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gk5\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\") pod \"e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") " pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.549639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.549599    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:08.786821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.786714    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"
Oct 30 22:13:08.786821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.786760    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx" event={"ID":"966ea468-60a5-4c66-8243-a008d56cca2a","Type":"ContainerDied","Data":"edd73fc760df04214951812d96b039d6eef1d2b51f3931d50f3dea6649687223"}
Oct 30 22:13:08.787030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.786837    5510 scope.go:122] "RemoveContainer" containerID="15caab9d76829ddd2ee7c36a9a2a23d7b19967e782cd2363db8bd770719cc7e5"
Oct 30 22:13:08.942524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.942483    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:13:08.946319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.946294    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d434591-5d5665c886-l6prx"]
Oct 30 22:13:08.991845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:08.991811    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:13:09.000674 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:13:09.000639    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2b331c_a9a6_42b0_93df_8465432bd116.slice/crio-be39796db8576280e7dfe235c18bae998d80adc3c0d7c6db471ad17916855960 WatchSource:0}: Error finding container be39796db8576280e7dfe235c18bae998d80adc3c0d7c6db471ad17916855960: Status 404 returned error can't find the container with id be39796db8576280e7dfe235c18bae998d80adc3c0d7c6db471ad17916855960
Oct 30 22:13:09.003792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.003746    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:09.003893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.003833    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:09.003893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.003870    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:09.133618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.133578    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966ea468-60a5-4c66-8243-a008d56cca2a" path="/var/lib/kubelet/pods/966ea468-60a5-4c66-8243-a008d56cca2a/volumes"
Oct 30 22:13:09.796990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.796952    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerStarted","Data":"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"}
Oct 30 22:13:09.797163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.797010    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerStarted","Data":"be39796db8576280e7dfe235c18bae998d80adc3c0d7c6db471ad17916855960"}
Oct 30 22:13:09.910084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:09.910025    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podStartSLOduration=2.910010405 podStartE2EDuration="2.910010405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:09.90713374 +0000 UTC m=+1798.891040616" watchObservedRunningTime="2026-10-30 22:13:09.910010405 +0000 UTC m=+1798.893917282"
Oct 30 22:13:10.314912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:10.314880    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.550542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:10.550509    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:10.556314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:10.556285    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:10.804856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:10.804824    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:10.805709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:10.805685    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:13:11.159723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:11.159651    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:11.680370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:11.680341    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:13:11.907466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:11.904901    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.181307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:12.181273    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:13:12.484219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:12.484187    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.798664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:12.798589    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.978983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:12.978932    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:13.284690 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:13:13.284651    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:13:15.007146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.007108    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:13:15.012890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.012863    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.062650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.062604    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e66edb-serving-cert\""
Oct 30 22:13:15.072898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.072869    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:13:15.125290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.125261    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.125470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.125310    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zzm\" (UniqueName: \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.125516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.125463    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.125735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.125712    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.227232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.227198    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.227412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.227277    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.227412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.227380    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zzm\" (UniqueName: \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.227554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.227528    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.227860 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:13:15.227767    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-serving-cert: secret "maas-api-e2e-shared-b-w2-e66edb-serving-cert" not found
Oct 30 22:13:15.228054 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:13:15.228036    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls podName:70e0c9b4-9c4f-4481-b490-1576fede6f14 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:15.728017878 +0000 UTC m=+1804.711924735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls") pod "maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" (UID: "70e0c9b4-9c4f-4481-b490-1576fede6f14") : secret "maas-api-e2e-shared-b-w2-e66edb-serving-cert" not found
Oct 30 22:13:15.228403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.228381    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.229948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.229918    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.232905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.232884    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.298666 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.298540    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zzm\" (UniqueName: \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.474244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.474208    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.641085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.641006    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:15.750500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.750465    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.752850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.752822    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.926992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.926910    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:15.942539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:15.942511    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.180696 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.180559    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.349817 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.347538    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:13:16.367046 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:13:16.363975    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e0c9b4_9c4f_4481_b490_1576fede6f14.slice/crio-fca8567ef8ca2853100a9194c33e717d430d850a318b3a6a794b0e7b0d248030 WatchSource:0}: Error finding container fca8567ef8ca2853100a9194c33e717d430d850a318b3a6a794b0e7b0d248030: Status 404 returned error can't find the container with id fca8567ef8ca2853100a9194c33e717d430d850a318b3a6a794b0e7b0d248030
Oct 30 22:13:16.689617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.686733    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:13:16.695957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.695359    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.696557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.695260    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:13:16.707908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.707879    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hfhx\" (UniqueName: \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.708132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.708115    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.762655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.762631    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-c9dc10-dockercfg-rl4hw\""
Oct 30 22:13:16.763513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.763493    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-c9dc10\""
Oct 30 22:13:16.809955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.809925    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hfhx\" (UniqueName: \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.810170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.810150    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.810843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.810826    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.864170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.864146    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hfhx\" (UniqueName: \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\") pod \"payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:16.904760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:16.904727    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" event={"ID":"70e0c9b4-9c4f-4481-b490-1576fede6f14","Type":"ContainerStarted","Data":"fca8567ef8ca2853100a9194c33e717d430d850a318b3a6a794b0e7b0d248030"}
Oct 30 22:13:17.016186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.016152    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:17.409207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.409175    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:13:17.415134 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:13:17.415092    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0920ec87_5acb_4457_80b5_1c8479e06c07.slice/crio-39f8193fabe02e2381acba74d232f2b3bfa84a3130f283f586b93a46a55c1bdc WatchSource:0}: Error finding container 39f8193fabe02e2381acba74d232f2b3bfa84a3130f283f586b93a46a55c1bdc: Status 404 returned error can't find the container with id 39f8193fabe02e2381acba74d232f2b3bfa84a3130f283f586b93a46a55c1bdc
Oct 30 22:13:17.914874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.914788    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" event={"ID":"70e0c9b4-9c4f-4481-b490-1576fede6f14","Type":"ContainerStarted","Data":"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"}
Oct 30 22:13:17.914874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.914862    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:17.916413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.916388    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" event={"ID":"0920ec87-5acb-4457-80b5-1c8479e06c07","Type":"ContainerStarted","Data":"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"}
Oct 30 22:13:17.916546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.916440    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" event={"ID":"0920ec87-5acb-4457-80b5-1c8479e06c07","Type":"ContainerStarted","Data":"39f8193fabe02e2381acba74d232f2b3bfa84a3130f283f586b93a46a55c1bdc"}
Oct 30 22:13:17.916546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:17.916468    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:18.026892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:18.026839    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" podStartSLOduration=3.143028045 podStartE2EDuration="4.02682559s" totalImagesPullingTime="883.797545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:18.024388496 +0000 UTC m=+1807.008295374" watchObservedRunningTime="2026-10-30 22:13:18.02682559 +0000 UTC m=+1807.010732466"
Oct 30 22:13:18.044667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:18.044635    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.146763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:18.146698    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" podStartSLOduration=2.146683701 podStartE2EDuration="2.146683701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:18.145993426 +0000 UTC m=+1807.129900323" watchObservedRunningTime="2026-10-30 22:13:18.146683701 +0000 UTC m=+1807.130590576"
Oct 30 22:13:18.221493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:18.221461    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.393806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:18.391605    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:20.664192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:20.664136    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:22.053601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.053566    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"]
Oct 30 22:13:22.098818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.098773    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76051f8a-vcszq"]
Oct 30 22:13:22.806312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.806280    5510 scope.go:122] "RemoveContainer" containerID="76bc34728dcdd96104ecb421404b0a3bcb3527a500df10e81385a0816ec4bc98"
Oct 30 22:13:22.816474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.816455    5510 scope.go:122] "RemoveContainer" containerID="c8d3e8b4cd8ea28101a6c608c5bf969b35ee9dca7d989fa662e0e5f89948d663"
Oct 30 22:13:22.825850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.825833    5510 scope.go:122] "RemoveContainer" containerID="44dfcf31534480a3c6566b51bd499a546186516174ba7fba0c6558638fe91748"
Oct 30 22:13:22.834464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:22.834437    5510 scope.go:122] "RemoveContainer" containerID="d420e1c74e8010b7c9888b2adf634fedd9aa6b4d1ee3603f1dcc12716bf8e8eb"
Oct 30 22:13:23.066203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:23.066121    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:23.359930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:23.352387    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:23.365525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:23.365485    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b4f0ff-d5e9-4048-b49f-659a8233154b" path="/var/lib/kubelet/pods/21b4f0ff-d5e9-4048-b49f-659a8233154b/volumes"
Oct 30 22:13:23.945499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:23.945472    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:13:25.455768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:25.455736    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.260540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:27.260504    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.449478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:27.449445    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.172678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:28.172625    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.944346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:28.944311    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:13:29.918602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:29.918568    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.303685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:34.303651    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.549811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:34.549608    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:34.760678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:34.760640    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:38.920091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:38.920053    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:42.840355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:42.840323    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:43.283668 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:13:43.283628    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:13:44.323413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:44.323378    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:44.494145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:44.494116    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:44.700440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:44.700357    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.246369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:45.246323    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:49.259833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:49.259796    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:51.454571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:51.454532    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:51.623102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:51.623062    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:51.849886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:51.849830    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:53.104428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:53.104401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:53.816804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:53.814649    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:53.987816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:53.987747    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:13:53.993806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:53.993769    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.084863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.084766    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:13:54.096012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.095974    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-263513-openshift-default-dockercfg-fzqcl\""
Oct 30 22:13:54.099965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.099933    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:54.248048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.248013    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.248629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.248603    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.248824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.248807    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.248965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.248951    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.249175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.249157    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kffwt\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.249317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.249303    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.249428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.249415    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.249539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.249525    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.249703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.249689    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.350932    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351000    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351083    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351126    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351220    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kffwt\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351271    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351306    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351369    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351444    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351591    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.351652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.351590    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.352246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.352224    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.352444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.352424    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.352482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.352455    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.353727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.353703    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.354464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.354445    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.410931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.410773    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:54.410931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.410810    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kffwt\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.417240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.417185    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\") pod \"e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") " pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:54.609602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:54.609516    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:55.061199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.061168    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:13:55.064467 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:13:55.064437    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd46d7c_c737_41e2_a61d_f36e297bb032.slice/crio-692660f0c5e61f5153ecbf01f27e892b25bcc39ba2eeab9fd805c54ac8fcbf9a WatchSource:0}: Error finding container 692660f0c5e61f5153ecbf01f27e892b25bcc39ba2eeab9fd805c54ac8fcbf9a: Status 404 returned error can't find the container with id 692660f0c5e61f5153ecbf01f27e892b25bcc39ba2eeab9fd805c54ac8fcbf9a
Oct 30 22:13:55.067636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.067596    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:55.067732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.067688    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:55.067792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.067727    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:13:55.284499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.284452    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerStarted","Data":"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"}
Oct 30 22:13:55.284947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.284560    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerStarted","Data":"692660f0c5e61f5153ecbf01f27e892b25bcc39ba2eeab9fd805c54ac8fcbf9a"}
Oct 30 22:13:55.401936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.401832    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podStartSLOduration=2.40181306 podStartE2EDuration="2.40181306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:55.395984665 +0000 UTC m=+1844.379891544" watchObservedRunningTime="2026-10-30 22:13:55.40181306 +0000 UTC m=+1844.385719939"
Oct 30 22:13:55.611808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:55.611272    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:56.616438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:56.616406    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:57.300668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:57.300637    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:57.301675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:57.301653    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:13:58.486689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:58.486646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:58.673881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:58.662351    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:58.956236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:58.956153    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:59.131812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:59.131758    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:59.313404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:13:59.313361    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:02.054211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:02.054169    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:02.244906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:02.240062    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:03.489803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:03.484346    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:08.298327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:08.298287    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:08.740647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:08.740613    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:14:12.835651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:12.835607    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:14:12.836171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:12.835948    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy" containerID="cri-o://293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711" gracePeriod=30
Oct 30 22:14:12.886761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:12.886701    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:14:12.896161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:12.896118    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy" containerID="cri-o://bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a" gracePeriod=30
Oct 30 22:14:13.044068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:13.044037    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:13.330022 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:13.329988    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:14:17.890551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:17.890479    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:18.251980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.250996    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:14:18.286891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.286794    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:14:18.362934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.362900    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.362989    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363063    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363184    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363258    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363316    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363427    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363479    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363513    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363551    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363585    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363601    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.363711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363731    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363774    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\") pod \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\" (UID: \"21c7bb10-d5a0-4326-971c-ade1c0c1f046\") "
Oct 30 22:14:18.363974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363852    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.363974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.363922    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\") pod \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\" (UID: \"4bbb86f1-2ca4-49d1-b20a-6213aad11600\") "
Oct 30 22:14:18.364198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364174    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:18.364328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364302    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:18.364429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364402    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.365094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364729    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.365094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364758    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.365094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364775    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.365094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.364806    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.366085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.365764    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.366085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.365838    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.366085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.365935    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.366085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.365953    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.366367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.366087    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "kube-api-access-ntb5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.366514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.366477    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.366662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.366634    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.367964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.367933    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.368070 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.368043    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:18.368170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.368149    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.368270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.368255    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.368967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.368934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs" pod "21c7bb10-d5a0-4326-971c-ade1c0c1f046" (UID: "21c7bb10-d5a0-4326-971c-ade1c0c1f046"). InnerVolumeSpecName "kube-api-access-vllfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:18.369382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.369358    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:18.369622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.369604    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo" pod "4bbb86f1-2ca4-49d1-b20a-6213aad11600" (UID: "4bbb86f1-2ca4-49d1-b20a-6213aad11600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471503    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471532    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471542    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllfs\" (UniqueName: \"kubernetes.io/projected/21c7bb10-d5a0-4326-971c-ade1c0c1f046-kube-api-access-vllfs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471552    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471561    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471572    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471581    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471590    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntb5r\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-kube-api-access-ntb5r\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471598    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471606    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471614    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471622    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbb86f1-2ca4-49d1-b20a-6213aad11600-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471630    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.476826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.471640    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c7bb10-d5a0-4326-971c-ade1c0c1f046-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:18.717085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.717049    5510 generic.go:356] "Generic (PLEG): container finished" podID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerID="bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a" exitCode=0
Oct 30 22:14:18.717272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.717114    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"
Oct 30 22:14:18.717272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.717139    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" event={"ID":"4bbb86f1-2ca4-49d1-b20a-6213aad11600","Type":"ContainerDied","Data":"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"}
Oct 30 22:14:18.717272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.717200    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d" event={"ID":"4bbb86f1-2ca4-49d1-b20a-6213aad11600","Type":"ContainerDied","Data":"fbd91e485c6e3edb848b57c4efc59d7cdca3979bd89cfc75644475f081d59f33"}
Oct 30 22:14:18.717272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.717233    5510 scope.go:122] "RemoveContainer" containerID="bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"
Oct 30 22:14:18.718697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.718676    5510 generic.go:356] "Generic (PLEG): container finished" podID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerID="293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711" exitCode=0
Oct 30 22:14:18.718871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.718851    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" event={"ID":"21c7bb10-d5a0-4326-971c-ade1c0c1f046","Type":"ContainerDied","Data":"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"}
Oct 30 22:14:18.718943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.718899    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm" event={"ID":"21c7bb10-d5a0-4326-971c-ade1c0c1f046","Type":"ContainerDied","Data":"3be860c30673cc344dea3a6af51a7792e311e6cccd4b1e8f9e6fbeef653025ee"}
Oct 30 22:14:18.719017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.718978    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"
Oct 30 22:14:18.728249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.728198    5510 scope.go:122] "RemoveContainer" containerID="bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"
Oct 30 22:14:18.728798 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:18.728480    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a\": container with ID starting with bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a not found: ID does not exist" containerID="bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"
Oct 30 22:14:18.728798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.728507    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a"} err="failed to get container status \"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a\": rpc error: code = NotFound desc = could not find container \"bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a\": container with ID starting with bb938a9bc7dfe9efba92ea3e73701fc8f4363de93d2e937ada237db2d50efd3a not found: ID does not exist"
Oct 30 22:14:18.728798 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.728529    5510 scope.go:122] "RemoveContainer" containerID="293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"
Oct 30 22:14:18.738685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.738668    5510 scope.go:122] "RemoveContainer" containerID="293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"
Oct 30 22:14:18.738983 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:18.738965    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711\": container with ID starting with 293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711 not found: ID does not exist" containerID="293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"
Oct 30 22:14:18.739082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.738989    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711"} err="failed to get container status \"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711\": rpc error: code = NotFound desc = could not find container \"293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711\": container with ID starting with 293dbe7d7cda00dd0a7126346e949f8709779690d026b8b539746bad9be29711 not found: ID does not exist"
Oct 30 22:14:18.875804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.874270    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:14:18.877582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.877556    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-x8gnm"]
Oct 30 22:14:18.983957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.983874    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:14:18.989023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:18.988990    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-26m5d"]
Oct 30 22:14:19.130291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.130260    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" path="/var/lib/kubelet/pods/21c7bb10-d5a0-4326-971c-ade1c0c1f046/volumes"
Oct 30 22:14:19.130843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.130827    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" path="/var/lib/kubelet/pods/4bbb86f1-2ca4-49d1-b20a-6213aad11600/volumes"
Oct 30 22:14:19.747049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.747018    5510 generic.go:356] "Generic (PLEG): container finished" podID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerID="461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737" exitCode=0
Oct 30 22:14:19.747234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.747087    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerDied","Data":"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"}
Oct 30 22:14:19.747520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.747502    5510 scope.go:122] "RemoveContainer" containerID="461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"
Oct 30 22:14:19.749242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.749208    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:19.749331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.749278    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:19.749331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:19.749309    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:20.783386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.783340    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerStarted","Data":"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"}
Oct 30 22:14:20.789833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.789798    5510 generic.go:356] "Generic (PLEG): container finished" podID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerID="f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875" exitCode=0
Oct 30 22:14:20.790095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.790075    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerDied","Data":"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"}
Oct 30 22:14:20.790706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.790686    5510 scope.go:122] "RemoveContainer" containerID="f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"
Oct 30 22:14:20.793987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.793530    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:20.793987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.793625    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:20.793987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:20.793674    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:14:21.785805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:21.782042    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:14:21.785805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:21.782197    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:14:21.785805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:21.782258    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:14:21.803476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:21.803435    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerStarted","Data":"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"}
Oct 30 22:14:22.248062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.248026    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:14:22.248306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.248283    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:22.248360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.248336    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:22.331602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.331543    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:22.784802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.784174    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:14:22.784802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.784240    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:14:22.889979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:22.889938    5510 scope.go:122] "RemoveContainer" containerID="96ea9676c377918a2560fa2207019fc893ddf2e0e9465d52fcf3fa58c57772ec"
Oct 30 22:14:23.248547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:23.248515    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:23.248729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:23.248571    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:23.781803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:23.781188    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:14:23.781803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:23.781266    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:14:24.248761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:24.248724    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:24.249204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:24.248830    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:24.789803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:24.789160    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:14:24.789803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:24.789221    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:14:25.248435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:25.248401    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:25.248614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:25.248454    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:25.785807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:25.783177    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 22:14:25.785807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:25.783244    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 22:14:26.248423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.248381    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:26.248625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.248463    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:26.774255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.774221    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:26.788888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.788853    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:14:26.789335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.789310    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:14:26.790079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:26.790057    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:14:27.248389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:27.248356    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:14:27.248567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:27.248404    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 22:14:27.595888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:27.595802    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:28.253479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:28.253447    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:14:28.253956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:28.253905    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:14:28.254607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:28.254589    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:14:31.223135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:31.220713    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:31.400322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:31.400293    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.536428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:35.536396    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.693908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:35.693877    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.918549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:35.918463    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:39.909275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.909237    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:14:39.910053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910037    5510 cpu_manager.go:414] "containerMap: removing container" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy"
Oct 30 22:14:39.910120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910054    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy"
Oct 30 22:14:39.910120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910070    5510 cpu_manager.go:414] "containerMap: removing container" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy"
Oct 30 22:14:39.910120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910075    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy"
Oct 30 22:14:39.910218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910195    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c7bb10-d5a0-4326-971c-ade1c0c1f046" containerName="istio-proxy"
Oct 30 22:14:39.910218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.910211    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbb86f1-2ca4-49d1-b20a-6213aad11600" containerName="istio-proxy"
Oct 30 22:14:39.914346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.914327    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:39.967010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:39.966983    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:14:40.008338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.008306    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-25de17-serving-cert\""
Oct 30 22:14:40.020655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.020622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.020824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.020807    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.020917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.020893    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnn4x\" (UniqueName: \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.021024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.021011    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.122191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.122154    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.122376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.122245    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.122376 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:40.122298    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-serving-cert: secret "maas-api-e2e-api-a-25de17-serving-cert" not found
Oct 30 22:14:40.122376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.122343    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.122554 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:40.122377    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls podName:f3efe962-3a6f-4529-b28c-c4f363de96e3 nodeName:}" failed. No retries permitted until 2026-10-30 22:14:40.622356322 +0000 UTC m=+1889.606263192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls") pod "maas-api-e2e-api-a-25de17-f888876bc-crksv" (UID: "f3efe962-3a6f-4529-b28c-c4f363de96e3") : secret "maas-api-e2e-api-a-25de17-serving-cert" not found
Oct 30 22:14:40.122554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.122438    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnn4x\" (UniqueName: \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.122934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.122915    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.124891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.124874    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.132580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.132561    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.182597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.182481    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnn4x\" (UniqueName: \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.315270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.315192    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.484602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.484567    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:40.632305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.632273    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.634755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.634725    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") pod \"maas-api-e2e-api-a-25de17-f888876bc-crksv\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:40.828694 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:40.828606    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:41.205491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:41.205403    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:14:41.212593 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:14:41.212564    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3efe962_3a6f_4529_b28c_c4f363de96e3.slice/crio-c141d8bfe851cebd6fb9402c80cb3079cdebb57efa9cec3c8f88ba48f8f804be WatchSource:0}: Error finding container c141d8bfe851cebd6fb9402c80cb3079cdebb57efa9cec3c8f88ba48f8f804be: Status 404 returned error can't find the container with id c141d8bfe851cebd6fb9402c80cb3079cdebb57efa9cec3c8f88ba48f8f804be
Oct 30 22:14:42.016148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:42.016110    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" event={"ID":"f3efe962-3a6f-4529-b28c-c4f363de96e3","Type":"ContainerStarted","Data":"c141d8bfe851cebd6fb9402c80cb3079cdebb57efa9cec3c8f88ba48f8f804be"}
Oct 30 22:14:43.025434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:43.025398    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" event={"ID":"f3efe962-3a6f-4529-b28c-c4f363de96e3","Type":"ContainerStarted","Data":"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"}
Oct 30 22:14:43.025953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:43.025536    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:43.140633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:43.140582    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" podStartSLOduration=3.22530538 podStartE2EDuration="4.140568468s" totalImagesPullingTime="915.263088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:43.138319535 +0000 UTC m=+1892.122226443" watchObservedRunningTime="2026-10-30 22:14:43.140568468 +0000 UTC m=+1892.124475373"
Oct 30 22:14:43.334502 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:14:43.334418    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:14:44.659677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:44.659646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:49.038418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:49.038385    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:14:49.374580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:49.374467    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:54.219588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:54.219497    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:59.135146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:14:59.135106    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.210425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.210389    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:00.215747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.215723    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.238521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.238498    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:00.243055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.243038    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.314799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.311596    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.314799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.311705    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplbt\" (UniqueName: \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.314799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.311793    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.314799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.311959    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmf88\" (UniqueName: \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.314799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.312141    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:00.318899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.318031    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:00.318899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.318150    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.347359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.347327    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:00.347515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.347387    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:00.351750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.351718    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:15:00.380805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.366003    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.438993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.438962    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.477914    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479165    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvpr\" (UniqueName: \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479256    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479288    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479372    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmf88\" (UniqueName: \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479456    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4prk\" (UniqueName: \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479546    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479663    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplbt\" (UniqueName: \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.489865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.479753    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.523060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.522840    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.523326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.523174    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.544602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.544563    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:00.545120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.544856    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.549955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.549939    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:00.550177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.550163    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:00.550343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.550329    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:00.551568 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.550498    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:00.558993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.550117    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.571812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.571763    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:00.572043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.572025    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:00.572176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.572164    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:00.572278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.572268    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:00.572461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.572447    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.620137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.597021    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmf88\" (UniqueName: \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.620137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.599562    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplbt\" (UniqueName: \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.633030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.632989    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.633170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.633045    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.633170 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.633160    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4prk\" (UniqueName: \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.633308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.633211    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.633308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.633245    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8hq\" (UniqueName: \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.633647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.633491    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvpr\" (UniqueName: \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.636712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.634858    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.636858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.635433    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.717401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.717374    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4prk\" (UniqueName: \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\") pod \"maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.718132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.718109    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvpr\" (UniqueName: \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\") pod \"maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.738178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738151    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.738278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738187    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8hq\" (UniqueName: \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.738278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738228    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.738365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738324    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldk2q\" (UniqueName: \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.738401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738381    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4g8c\" (UniqueName: \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.738436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738405    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.738436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738423    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjzb\" (UniqueName: \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.738526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.738503    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.739341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.739317    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.815910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.814572    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8hq\" (UniqueName: \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\") pod \"maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.831316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.831284    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:00.839686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.839655    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldk2q\" (UniqueName: \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.839841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.839743    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4g8c\" (UniqueName: \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.839841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.839808    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.839841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.839837    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjzb\" (UniqueName: \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.840034 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.839946    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.840092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.840037    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.840913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.840884    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.841006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.840886    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.841629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.841609    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.848573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.848554    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:00.915802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.913241    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:00.917801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.916618    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:00.921989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.921950    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjzb\" (UniqueName: \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\") pod \"maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:00.922869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.922839    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4g8c\" (UniqueName: \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:00.923455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.923428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldk2q\" (UniqueName: \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\") pod \"maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:00.946273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.946143    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:00.949385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:00.949357    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:01.022243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:01.022209    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:01.050357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:01.050325    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:01.640167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:01.638569    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:02.020923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.020889    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:02.022112 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:15:02.021585    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f19f275_fbb6_4ee5_ad16_71eab77cddd4.slice/crio-470c122ca789cdc6f28c7425df023ffa55d3717fba5472d263badd1ef3152ed2 WatchSource:0}: Error finding container 470c122ca789cdc6f28c7425df023ffa55d3717fba5472d263badd1ef3152ed2: Status 404 returned error can't find the container with id 470c122ca789cdc6f28c7425df023ffa55d3717fba5472d263badd1ef3152ed2
Oct 30 22:15:02.047210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.047169    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:02.142164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.142133    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:02.202029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.201989    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:02.244030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.243951    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:02.249407 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:15:02.249371    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb050e62c_6750_4153_bd02_1c66ac2dfcee.slice/crio-3aa90fc6b789488730d146072872e71e51d943ab71602968c44cf6fef13926e2 WatchSource:0}: Error finding container 3aa90fc6b789488730d146072872e71e51d943ab71602968c44cf6fef13926e2: Status 404 returned error can't find the container with id 3aa90fc6b789488730d146072872e71e51d943ab71602968c44cf6fef13926e2
Oct 30 22:15:02.249508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.249405    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:02.324258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.324228    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:02.523674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.523637    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerStarted","Data":"5a42172f215087353022fcc8cfb03f0e8d96bbc6bd2a9cc8667447b097199c64"}
Oct 30 22:15:02.525708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.525677    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerStarted","Data":"a865e5dc2f87a50d7ec75ebb5ef9a0397f2735122e652cca5545fbc2d11023b2"}
Oct 30 22:15:02.528994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.528922    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerStarted","Data":"3aa90fc6b789488730d146072872e71e51d943ab71602968c44cf6fef13926e2"}
Oct 30 22:15:02.534232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.531711    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerStarted","Data":"6c52be2657997b53f855f2583ee93768d1036308ef07e6f5cef63f18afd3f2a9"}
Oct 30 22:15:02.534232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.531759    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerStarted","Data":"8d79af6df35c291518b8efa6a4d4c611805eb5a4d9723bd6df9d9c2daf8c078b"}
Oct 30 22:15:02.538816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.538750    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerStarted","Data":"1f168c579f41ff77095f7ad75d846586b5b07e3879f12b5014fc5ef7dc5e33d1"}
Oct 30 22:15:02.540514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.540481    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerStarted","Data":"7afdd8556d649ed25339cdd0957cc489597a809eda957f23a58fd4db1a5c43f6"}
Oct 30 22:15:02.568260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.546629    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerStarted","Data":"23df3f23a408af37723fde08f8e4ad9f463958512c14c2b8033375392186a5d1"}
Oct 30 22:15:02.568260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.547965    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerStarted","Data":"470c122ca789cdc6f28c7425df023ffa55d3717fba5472d263badd1ef3152ed2"}
Oct 30 22:15:02.648043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.647991    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" podStartSLOduration=2.647964541 podStartE2EDuration="2.647964541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.644234251 +0000 UTC m=+1911.628141124" watchObservedRunningTime="2026-10-30 22:15:02.647964541 +0000 UTC m=+1911.631871417"
Oct 30 22:15:02.900839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:02.900572    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:03.605159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.605121    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerStarted","Data":"a9848a1d55624a988790935713c1fdcfd4d720390328b972fd8a6d47b8e53fe0"}
Oct 30 22:15:03.608254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.608223    5510 generic.go:356] "Generic (PLEG): container finished" podID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerID="0253a6b80542f34b341d31a52e2342b36b9e9df26865232d8da8fbaf30fafe16" exitCode=60
Oct 30 22:15:03.608413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.608309    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerDied","Data":"0253a6b80542f34b341d31a52e2342b36b9e9df26865232d8da8fbaf30fafe16"}
Oct 30 22:15:03.608919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.608897    5510 scope.go:122] "RemoveContainer" containerID="0253a6b80542f34b341d31a52e2342b36b9e9df26865232d8da8fbaf30fafe16"
Oct 30 22:15:03.610892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.610867    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerStarted","Data":"1bd61d7357f2b93ebbb366242bc3a121011ef8f4cd555f15f65d40ba8042ec01"}
Oct 30 22:15:03.611135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.611115    5510 scope.go:122] "RemoveContainer" containerID="1bd61d7357f2b93ebbb366242bc3a121011ef8f4cd555f15f65d40ba8042ec01"
Oct 30 22:15:03.613184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.613161    5510 generic.go:356] "Generic (PLEG): container finished" podID="f4453024-52ad-45b5-9596-f6a9145f598b" containerID="fcc6b8286174f82e68f05ed07756e3220b4788d3be19d1cee2c94fc1d2b60c41" exitCode=60
Oct 30 22:15:03.613637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.613404    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerDied","Data":"fcc6b8286174f82e68f05ed07756e3220b4788d3be19d1cee2c94fc1d2b60c41"}
Oct 30 22:15:03.613637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.613542    5510 scope.go:122] "RemoveContainer" containerID="fcc6b8286174f82e68f05ed07756e3220b4788d3be19d1cee2c94fc1d2b60c41"
Oct 30 22:15:03.620071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.615727    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerStarted","Data":"e78d42064f4f1b58cb34c0481f2e1ba06a8c460ce017b9579ed63abfe8019bcb"}
Oct 30 22:15:03.620071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.616048    5510 scope.go:122] "RemoveContainer" containerID="e78d42064f4f1b58cb34c0481f2e1ba06a8c460ce017b9579ed63abfe8019bcb"
Oct 30 22:15:03.624054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.624029    5510 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerID="6c52be2657997b53f855f2583ee93768d1036308ef07e6f5cef63f18afd3f2a9" exitCode=60
Oct 30 22:15:03.624291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.624269    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerDied","Data":"6c52be2657997b53f855f2583ee93768d1036308ef07e6f5cef63f18afd3f2a9"}
Oct 30 22:15:03.624406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.624385    5510 scope.go:122] "RemoveContainer" containerID="6c52be2657997b53f855f2583ee93768d1036308ef07e6f5cef63f18afd3f2a9"
Oct 30 22:15:03.625988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.625959    5510 generic.go:356] "Generic (PLEG): container finished" podID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerID="b473ba81163a8b6e6ae766a8e3e03092222d8945741c3d056ae8062323f2ba8f" exitCode=60
Oct 30 22:15:03.626116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.626058    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerDied","Data":"b473ba81163a8b6e6ae766a8e3e03092222d8945741c3d056ae8062323f2ba8f"}
Oct 30 22:15:03.626341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.626326    5510 scope.go:122] "RemoveContainer" containerID="b473ba81163a8b6e6ae766a8e3e03092222d8945741c3d056ae8062323f2ba8f"
Oct 30 22:15:03.627682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.627660    5510 generic.go:356] "Generic (PLEG): container finished" podID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerID="23bc4af913ec13e2b041447b104483a63d052a9b3c4a6598335ab0aa2469847a" exitCode=60
Oct 30 22:15:03.627774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.627737    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerDied","Data":"23bc4af913ec13e2b041447b104483a63d052a9b3c4a6598335ab0aa2469847a"}
Oct 30 22:15:03.627918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.627904    5510 scope.go:122] "RemoveContainer" containerID="23bc4af913ec13e2b041447b104483a63d052a9b3c4a6598335ab0aa2469847a"
Oct 30 22:15:03.740189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:03.740138    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" podStartSLOduration=3.740120435 podStartE2EDuration="3.740120435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.735737975 +0000 UTC m=+1912.719644854" watchObservedRunningTime="2026-10-30 22:15:03.740120435 +0000 UTC m=+1912.724027313"
Oct 30 22:15:04.025062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.025025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:04.639886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.639813    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerStarted","Data":"33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"}
Oct 30 22:15:04.642487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.642459    5510 generic.go:356] "Generic (PLEG): container finished" podID="89df3d06-8903-46e7-92ea-e278438fffbd" containerID="1bd61d7357f2b93ebbb366242bc3a121011ef8f4cd555f15f65d40ba8042ec01" exitCode=60
Oct 30 22:15:04.642647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.642550    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerDied","Data":"1bd61d7357f2b93ebbb366242bc3a121011ef8f4cd555f15f65d40ba8042ec01"}
Oct 30 22:15:04.642647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.642607    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerStarted","Data":"e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"}
Oct 30 22:15:04.645418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.644925    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerStarted","Data":"bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"}
Oct 30 22:15:04.647736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.647711    5510 generic.go:356] "Generic (PLEG): container finished" podID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerID="e78d42064f4f1b58cb34c0481f2e1ba06a8c460ce017b9579ed63abfe8019bcb" exitCode=60
Oct 30 22:15:04.647855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.647746    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerDied","Data":"e78d42064f4f1b58cb34c0481f2e1ba06a8c460ce017b9579ed63abfe8019bcb"}
Oct 30 22:15:04.647855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.647804    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerStarted","Data":"145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"}
Oct 30 22:15:04.649904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.649883    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerStarted","Data":"f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"}
Oct 30 22:15:04.652067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.652048    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerStarted","Data":"2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"}
Oct 30 22:15:04.657810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.655072    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerStarted","Data":"a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"}
Oct 30 22:15:04.662527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.662501    5510 generic.go:356] "Generic (PLEG): container finished" podID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerID="a9848a1d55624a988790935713c1fdcfd4d720390328b972fd8a6d47b8e53fe0" exitCode=60
Oct 30 22:15:04.662651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.662582    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerDied","Data":"a9848a1d55624a988790935713c1fdcfd4d720390328b972fd8a6d47b8e53fe0"}
Oct 30 22:15:04.662901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.662883    5510 scope.go:122] "RemoveContainer" containerID="a9848a1d55624a988790935713c1fdcfd4d720390328b972fd8a6d47b8e53fe0"
Oct 30 22:15:04.760939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.757661    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" podStartSLOduration=4.757641933 podStartE2EDuration="4.757641933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.755053891 +0000 UTC m=+1913.738960769" watchObservedRunningTime="2026-10-30 22:15:04.757641933 +0000 UTC m=+1913.741548810"
Oct 30 22:15:04.867445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.867381    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" podStartSLOduration=4.867360092 podStartE2EDuration="4.867360092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.864885191 +0000 UTC m=+1913.848792069" watchObservedRunningTime="2026-10-30 22:15:04.867360092 +0000 UTC m=+1913.851266969"
Oct 30 22:15:04.971376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:04.971313    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" podStartSLOduration=4.971294909 podStartE2EDuration="4.971294909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.970274709 +0000 UTC m=+1913.954181657" watchObservedRunningTime="2026-10-30 22:15:04.971294909 +0000 UTC m=+1913.955201786"
Oct 30 22:15:05.087742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.087681    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" podStartSLOduration=5.087666946 podStartE2EDuration="5.087666946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.086928139 +0000 UTC m=+1914.070835014" watchObservedRunningTime="2026-10-30 22:15:05.087666946 +0000 UTC m=+1914.071573853"
Oct 30 22:15:05.193394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.193337    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" podStartSLOduration=5.193319237 podStartE2EDuration="5.193319237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.192287507 +0000 UTC m=+1914.176194385" watchObservedRunningTime="2026-10-30 22:15:05.193319237 +0000 UTC m=+1914.177226114"
Oct 30 22:15:05.417642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.417577    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" podStartSLOduration=5.417557921 podStartE2EDuration="5.417557921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.413653184 +0000 UTC m=+1914.397560062" watchObservedRunningTime="2026-10-30 22:15:05.417557921 +0000 UTC m=+1914.401464808"
Oct 30 22:15:05.677508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.677412    5510 generic.go:356] "Generic (PLEG): container finished" podID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerID="33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb" exitCode=60
Oct 30 22:15:05.677683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.677497    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerDied","Data":"33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"}
Oct 30 22:15:05.677683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.677571    5510 scope.go:122] "RemoveContainer" containerID="0253a6b80542f34b341d31a52e2342b36b9e9df26865232d8da8fbaf30fafe16"
Oct 30 22:15:05.677933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.677907    5510 scope.go:122] "RemoveContainer" containerID="33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"
Oct 30 22:15:05.678632 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.678599    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg_odh-ai-gateway-infra(3f19f275-fbb6-4ee5-ad16-71eab77cddd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4"
Oct 30 22:15:05.680952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.680930    5510 generic.go:356] "Generic (PLEG): container finished" podID="89df3d06-8903-46e7-92ea-e278438fffbd" containerID="e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0" exitCode=60
Oct 30 22:15:05.681421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.681403    5510 scope.go:122] "RemoveContainer" containerID="e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"
Oct 30 22:15:05.681856 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.681824    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7_odh-ai-gateway-infra(89df3d06-8903-46e7-92ea-e278438fffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" podUID="89df3d06-8903-46e7-92ea-e278438fffbd"
Oct 30 22:15:05.681941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.681875    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerDied","Data":"e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"}
Oct 30 22:15:05.684593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.684567    5510 generic.go:356] "Generic (PLEG): container finished" podID="f4453024-52ad-45b5-9596-f6a9145f598b" containerID="bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b" exitCode=60
Oct 30 22:15:05.685125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.685107    5510 scope.go:122] "RemoveContainer" containerID="bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"
Oct 30 22:15:05.685296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.685274    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerDied","Data":"bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"}
Oct 30 22:15:05.685484 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.685458    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k_odh-ai-gateway-infra(f4453024-52ad-45b5-9596-f6a9145f598b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" podUID="f4453024-52ad-45b5-9596-f6a9145f598b"
Oct 30 22:15:05.689691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.687700    5510 generic.go:356] "Generic (PLEG): container finished" podID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerID="145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5" exitCode=60
Oct 30 22:15:05.689691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.688136    5510 scope.go:122] "RemoveContainer" containerID="145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"
Oct 30 22:15:05.689691 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.688463    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2_odh-ai-gateway-infra(b050e62c-6750-4153-bd02-1c66ac2dfcee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee"
Oct 30 22:15:05.689691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.688547    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerDied","Data":"145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"}
Oct 30 22:15:05.691979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.691952    5510 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerID="f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c" exitCode=60
Oct 30 22:15:05.692267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.692244    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerDied","Data":"f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"}
Oct 30 22:15:05.692480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.692463    5510 scope.go:122] "RemoveContainer" containerID="f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"
Oct 30 22:15:05.692909 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.692884    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb_odh-ai-gateway-infra(8c2c9b14-68a1-4f21-a5de-f10fd8053e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52"
Oct 30 22:15:05.695007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.694739    5510 generic.go:356] "Generic (PLEG): container finished" podID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerID="2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e" exitCode=60
Oct 30 22:15:05.695007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.694992    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerDied","Data":"2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"}
Oct 30 22:15:05.695408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.695387    5510 scope.go:122] "RemoveContainer" containerID="2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"
Oct 30 22:15:05.695759 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.695698    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77_odh-ai-gateway-infra(1e35b445-5371-4bbe-8d1c-a372b41daac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2"
Oct 30 22:15:05.698193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.698123    5510 generic.go:356] "Generic (PLEG): container finished" podID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerID="a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218" exitCode=60
Oct 30 22:15:05.698315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.698274    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerDied","Data":"a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"}
Oct 30 22:15:05.698621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.698602    5510 scope.go:122] "RemoveContainer" containerID="a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"
Oct 30 22:15:05.698984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.698926    5510 scope.go:122] "RemoveContainer" containerID="1bd61d7357f2b93ebbb366242bc3a121011ef8f4cd555f15f65d40ba8042ec01"
Oct 30 22:15:05.699081 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:05.698941    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8_odh-ai-gateway-infra(8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b"
Oct 30 22:15:05.701419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.701396    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerStarted","Data":"00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"}
Oct 30 22:15:05.715737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.715699    5510 scope.go:122] "RemoveContainer" containerID="fcc6b8286174f82e68f05ed07756e3220b4788d3be19d1cee2c94fc1d2b60c41"
Oct 30 22:15:05.733346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.733323    5510 scope.go:122] "RemoveContainer" containerID="e78d42064f4f1b58cb34c0481f2e1ba06a8c460ce017b9579ed63abfe8019bcb"
Oct 30 22:15:05.755769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.755652    5510 scope.go:122] "RemoveContainer" containerID="6c52be2657997b53f855f2583ee93768d1036308ef07e6f5cef63f18afd3f2a9"
Oct 30 22:15:05.780603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.780575    5510 scope.go:122] "RemoveContainer" containerID="b473ba81163a8b6e6ae766a8e3e03092222d8945741c3d056ae8062323f2ba8f"
Oct 30 22:15:05.802101 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:05.802072    5510 scope.go:122] "RemoveContainer" containerID="23bc4af913ec13e2b041447b104483a63d052a9b3c4a6598335ab0aa2469847a"
Oct 30 22:15:06.715235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.715196    5510 generic.go:356] "Generic (PLEG): container finished" podID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerID="00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4" exitCode=60
Oct 30 22:15:06.715423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.715282    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerDied","Data":"00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"}
Oct 30 22:15:06.715423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.715374    5510 scope.go:122] "RemoveContainer" containerID="a9848a1d55624a988790935713c1fdcfd4d720390328b972fd8a6d47b8e53fe0"
Oct 30 22:15:06.715767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.715743    5510 scope.go:122] "RemoveContainer" containerID="00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"
Oct 30 22:15:06.716295 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.716262    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg_odh-ai-gateway-infra(fa3fbdb7-1b93-4a94-b372-60523f6959a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4"
Oct 30 22:15:06.718140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.718122    5510 scope.go:122] "RemoveContainer" containerID="33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"
Oct 30 22:15:06.718539 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.718477    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg_odh-ai-gateway-infra(3f19f275-fbb6-4ee5-ad16-71eab77cddd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4"
Oct 30 22:15:06.720148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.720126    5510 scope.go:122] "RemoveContainer" containerID="e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"
Oct 30 22:15:06.720468 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.720439    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7_odh-ai-gateway-infra(89df3d06-8903-46e7-92ea-e278438fffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" podUID="89df3d06-8903-46e7-92ea-e278438fffbd"
Oct 30 22:15:06.723343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.722545    5510 scope.go:122] "RemoveContainer" containerID="bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"
Oct 30 22:15:06.723343 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.722927    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k_odh-ai-gateway-infra(f4453024-52ad-45b5-9596-f6a9145f598b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" podUID="f4453024-52ad-45b5-9596-f6a9145f598b"
Oct 30 22:15:06.724841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.724816    5510 scope.go:122] "RemoveContainer" containerID="145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"
Oct 30 22:15:06.725186 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.725158    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2_odh-ai-gateway-infra(b050e62c-6750-4153-bd02-1c66ac2dfcee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee"
Oct 30 22:15:06.726698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.726427    5510 scope.go:122] "RemoveContainer" containerID="f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"
Oct 30 22:15:06.726698 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.726644    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb_odh-ai-gateway-infra(8c2c9b14-68a1-4f21-a5de-f10fd8053e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52"
Oct 30 22:15:06.730720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.730692    5510 scope.go:122] "RemoveContainer" containerID="2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"
Oct 30 22:15:06.731066 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.731031    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77_odh-ai-gateway-infra(1e35b445-5371-4bbe-8d1c-a372b41daac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2"
Oct 30 22:15:06.733679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:06.733658    5510 scope.go:122] "RemoveContainer" containerID="a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"
Oct 30 22:15:06.734039 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:06.734016    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8_odh-ai-gateway-infra(8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b"
Oct 30 22:15:07.433101 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:07.433055    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:15:07.433579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:07.433126    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:15:07.745331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:07.745297    5510 scope.go:122] "RemoveContainer" containerID="00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"
Oct 30 22:15:07.745792 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:07.745755    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg_odh-ai-gateway-infra(fa3fbdb7-1b93-4a94-b372-60523f6959a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4"
Oct 30 22:15:08.728562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:08.728524    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:09.802454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.802422    5510 generic.go:356] "Generic (PLEG): container finished" podID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerID="c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38" exitCode=0
Oct 30 22:15:09.802950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.802502    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerDied","Data":"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"}
Oct 30 22:15:09.803097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.803064    5510 scope.go:122] "RemoveContainer" containerID="c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"
Oct 30 22:15:09.804654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.804636    5510 generic.go:356] "Generic (PLEG): container finished" podID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerID="5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49" exitCode=0
Oct 30 22:15:09.804752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.804700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerDied","Data":"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"}
Oct 30 22:15:09.805014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.804988    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:09.805064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.805056    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:09.805099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.805088    5510 scope.go:122] "RemoveContainer" containerID="5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"
Oct 30 22:15:09.805178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.805089    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:09.806676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.806644    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:09.806814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.806703    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:09.806814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:09.806732    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:15:10.805091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:10.805046    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:15:10.813838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:10.813800    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerStarted","Data":"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"}
Oct 30 22:15:10.815964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:10.815940    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerStarted","Data":"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"}
Oct 30 22:15:11.327210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.327178    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:15:11.327406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.327379    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:11.327483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.327429    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:11.550353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.550265    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:15:11.555802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.551848    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:11.555802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:11.551916    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:12.327550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:12.327507    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:12.327978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:12.327585    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:12.550268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:12.550231    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:12.550455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:12.550302    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:13.327801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.327696    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:13.327801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.327759    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:13.337566 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:13.337526    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:15:13.422985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.422928    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.557803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.550197    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:13.557803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.550257    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:13.981804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:13.978709    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:14.329311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:14.329232    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:14.329748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:14.329722    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:14.550613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:14.550578    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:14.550919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:14.550889    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:14.811725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:14.811687    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:15.334815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:15.332775    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:15.334815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:15.332925    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:15.553832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:15.553773    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:15.554138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:15.554105    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:16.122567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.120798    5510 scope.go:122] "RemoveContainer" containerID="145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"
Oct 30 22:15:16.328520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.328471    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 22:15:16.328837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.328804    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 22:15:16.410933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.410288    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.555112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.554220    5510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:15:16.555112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.554300    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:15:16.610803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.609711    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.882867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:16.882814    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerStarted","Data":"ea745707ff8b1565cd24cb9b8fddf0ab95b1f7f2f64a75e5a836e42fe2925c30"}
Oct 30 22:15:17.118299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.118270    5510 scope.go:122] "RemoveContainer" containerID="2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"
Oct 30 22:15:17.118504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.118433    5510 scope.go:122] "RemoveContainer" containerID="00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"
Oct 30 22:15:17.118586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.118568    5510 scope.go:122] "RemoveContainer" containerID="33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"
Oct 30 22:15:17.332681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.332652    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:15:17.333066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.333043    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:15:17.333702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.333683    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:15:17.557237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.557209    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:15:17.557956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.557932    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:15:17.559934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.559622    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:15:17.892997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.892966    5510 generic.go:356] "Generic (PLEG): container finished" podID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerID="ea745707ff8b1565cd24cb9b8fddf0ab95b1f7f2f64a75e5a836e42fe2925c30" exitCode=60
Oct 30 22:15:17.893175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.893047    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerDied","Data":"ea745707ff8b1565cd24cb9b8fddf0ab95b1f7f2f64a75e5a836e42fe2925c30"}
Oct 30 22:15:17.893175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.893124    5510 scope.go:122] "RemoveContainer" containerID="145a6216756c81e879403fa38e3a18f6c24b1ee78222c73aaf7fab493bd214f5"
Oct 30 22:15:17.893512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.893488    5510 scope.go:122] "RemoveContainer" containerID="ea745707ff8b1565cd24cb9b8fddf0ab95b1f7f2f64a75e5a836e42fe2925c30"
Oct 30 22:15:17.893957 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:17.893924    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2_odh-ai-gateway-infra(b050e62c-6750-4153-bd02-1c66ac2dfcee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee"
Oct 30 22:15:17.895616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.895588    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerStarted","Data":"18b97efdf7e672b8daeade2c2620fe9050d9b578b1e9aefe1c9bf9927bbdf5ef"}
Oct 30 22:15:17.897830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.897806    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerStarted","Data":"dfae2c8187e502785f425c8de0fd8cf207f9365a5700737ef14e4ae9d0acbaba"}
Oct 30 22:15:17.900032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:17.900014    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerStarted","Data":"ce110a0d59f23e2ee64e8f25975efd2a8030ef53613d473e6e3cc22486c70450"}
Oct 30 22:15:18.017154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.017076    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:18.061267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.061234    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:18.124857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.124828    5510 scope.go:122] "RemoveContainer" containerID="f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"
Oct 30 22:15:18.129988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.126244    5510 scope.go:122] "RemoveContainer" containerID="e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"
Oct 30 22:15:18.131660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.131629    5510 scope.go:122] "RemoveContainer" containerID="bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"
Oct 30 22:15:18.909977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.909901    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerStarted","Data":"82e37fe69802a1c19a9e02b31f380a53e18a4d889c8073bdc4c73c558e2f7a07"}
Oct 30 22:15:18.913656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.913633    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerStarted","Data":"6a72380adfadca9d0d2202dc5ea2832b33845658ba48fe4baf08aa3b32d9172b"}
Oct 30 22:15:18.915708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.915687    5510 generic.go:356] "Generic (PLEG): container finished" podID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerID="18b97efdf7e672b8daeade2c2620fe9050d9b578b1e9aefe1c9bf9927bbdf5ef" exitCode=60
Oct 30 22:15:18.915838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.915759    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerDied","Data":"18b97efdf7e672b8daeade2c2620fe9050d9b578b1e9aefe1c9bf9927bbdf5ef"}
Oct 30 22:15:18.915838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.915823    5510 scope.go:122] "RemoveContainer" containerID="2be9a76a14bb0d28e872d14ec45876067ac08d7121c14a1d3bbbd6d3f143d75e"
Oct 30 22:15:18.916139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.916119    5510 scope.go:122] "RemoveContainer" containerID="18b97efdf7e672b8daeade2c2620fe9050d9b578b1e9aefe1c9bf9927bbdf5ef"
Oct 30 22:15:18.916472 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:18.916440    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77_odh-ai-gateway-infra(1e35b445-5371-4bbe-8d1c-a372b41daac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2"
Oct 30 22:15:18.917861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.917842    5510 generic.go:356] "Generic (PLEG): container finished" podID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerID="dfae2c8187e502785f425c8de0fd8cf207f9365a5700737ef14e4ae9d0acbaba" exitCode=60
Oct 30 22:15:18.917961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.917910    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerDied","Data":"dfae2c8187e502785f425c8de0fd8cf207f9365a5700737ef14e4ae9d0acbaba"}
Oct 30 22:15:18.918287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.918263    5510 scope.go:122] "RemoveContainer" containerID="dfae2c8187e502785f425c8de0fd8cf207f9365a5700737ef14e4ae9d0acbaba"
Oct 30 22:15:18.918599 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:18.918570    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg_odh-ai-gateway-infra(fa3fbdb7-1b93-4a94-b372-60523f6959a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4"
Oct 30 22:15:18.920118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.920099    5510 generic.go:356] "Generic (PLEG): container finished" podID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerID="ce110a0d59f23e2ee64e8f25975efd2a8030ef53613d473e6e3cc22486c70450" exitCode=60
Oct 30 22:15:18.920281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.920156    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerDied","Data":"ce110a0d59f23e2ee64e8f25975efd2a8030ef53613d473e6e3cc22486c70450"}
Oct 30 22:15:18.920546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.920515    5510 scope.go:122] "RemoveContainer" containerID="ce110a0d59f23e2ee64e8f25975efd2a8030ef53613d473e6e3cc22486c70450"
Oct 30 22:15:18.920852 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:18.920822    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg_odh-ai-gateway-infra(3f19f275-fbb6-4ee5-ad16-71eab77cddd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4"
Oct 30 22:15:18.922448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.922421    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerStarted","Data":"3fb8fbf24ee22a2fce5f98d4e71aac24ce012637fe2f9919e2034442b157bcac"}
Oct 30 22:15:18.928912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.928897    5510 scope.go:122] "RemoveContainer" containerID="00d26a044158af38c966525b39301e1bf801cd70044bee5eb564abf300e77ac4"
Oct 30 22:15:18.941179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:18.941142    5510 scope.go:122] "RemoveContainer" containerID="33eb545d1c3ab0e935752a948f37b70b93222935b21bfcb153197c618f92babb"
Oct 30 22:15:19.058057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.058030    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:19.099291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.099265    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\" (UniqueName: \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\") pod \"b050e62c-6750-4153-bd02-1c66ac2dfcee\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") "
Oct 30 22:15:19.099426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.099315    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\") pod \"b050e62c-6750-4153-bd02-1c66ac2dfcee\" (UID: \"b050e62c-6750-4153-bd02-1c66ac2dfcee\") "
Oct 30 22:15:19.099673 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.099651    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca" pod "b050e62c-6750-4153-bd02-1c66ac2dfcee" (UID: "b050e62c-6750-4153-bd02-1c66ac2dfcee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.099912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.099898    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b050e62c-6750-4153-bd02-1c66ac2dfcee-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.101322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.101297    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr" pod "b050e62c-6750-4153-bd02-1c66ac2dfcee" (UID: "b050e62c-6750-4153-bd02-1c66ac2dfcee"). InnerVolumeSpecName "kube-api-access-tzvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.118391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.118369    5510 scope.go:122] "RemoveContainer" containerID="a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"
Oct 30 22:15:19.131970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.131948    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:19.202089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.202023    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvpr\" (UniqueName: \"kubernetes.io/projected/b050e62c-6750-4153-bd02-1c66ac2dfcee-kube-api-access-tzvpr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.263017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.262977    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:19.376787 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.376756    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:19.931884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.931853    5510 generic.go:356] "Generic (PLEG): container finished" podID="89df3d06-8903-46e7-92ea-e278438fffbd" containerID="3fb8fbf24ee22a2fce5f98d4e71aac24ce012637fe2f9919e2034442b157bcac" exitCode=60
Oct 30 22:15:19.932351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.931935    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerDied","Data":"3fb8fbf24ee22a2fce5f98d4e71aac24ce012637fe2f9919e2034442b157bcac"}
Oct 30 22:15:19.932351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.932004    5510 scope.go:122] "RemoveContainer" containerID="e072c8202e7a4373e039b8c7602e159611638c90afe241714374eaec8188fac0"
Oct 30 22:15:19.932462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.932428    5510 scope.go:122] "RemoveContainer" containerID="3fb8fbf24ee22a2fce5f98d4e71aac24ce012637fe2f9919e2034442b157bcac"
Oct 30 22:15:19.932977 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:19.932869    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7_odh-ai-gateway-infra(89df3d06-8903-46e7-92ea-e278438fffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" podUID="89df3d06-8903-46e7-92ea-e278438fffbd"
Oct 30 22:15:19.933965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.933945    5510 generic.go:356] "Generic (PLEG): container finished" podID="f4453024-52ad-45b5-9596-f6a9145f598b" containerID="82e37fe69802a1c19a9e02b31f380a53e18a4d889c8073bdc4c73c558e2f7a07" exitCode=60
Oct 30 22:15:19.934085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.934004    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerDied","Data":"82e37fe69802a1c19a9e02b31f380a53e18a4d889c8073bdc4c73c558e2f7a07"}
Oct 30 22:15:19.934399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.934382    5510 scope.go:122] "RemoveContainer" containerID="82e37fe69802a1c19a9e02b31f380a53e18a4d889c8073bdc4c73c558e2f7a07"
Oct 30 22:15:19.934702 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:19.934679    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k_odh-ai-gateway-infra(f4453024-52ad-45b5-9596-f6a9145f598b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" podUID="f4453024-52ad-45b5-9596-f6a9145f598b"
Oct 30 22:15:19.935469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.935451    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"
Oct 30 22:15:19.935563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.935463    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2" event={"ID":"b050e62c-6750-4153-bd02-1c66ac2dfcee","Type":"ContainerDied","Data":"3aa90fc6b789488730d146072872e71e51d943ab71602968c44cf6fef13926e2"}
Oct 30 22:15:19.937505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.937478    5510 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerID="6a72380adfadca9d0d2202dc5ea2832b33845658ba48fe4baf08aa3b32d9172b" exitCode=60
Oct 30 22:15:19.937591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.937542    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerDied","Data":"6a72380adfadca9d0d2202dc5ea2832b33845658ba48fe4baf08aa3b32d9172b"}
Oct 30 22:15:19.938085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.938064    5510 scope.go:122] "RemoveContainer" containerID="6a72380adfadca9d0d2202dc5ea2832b33845658ba48fe4baf08aa3b32d9172b"
Oct 30 22:15:19.938569 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:19.938539    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb_odh-ai-gateway-infra(8c2c9b14-68a1-4f21-a5de-f10fd8053e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52"
Oct 30 22:15:19.941684 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.941658    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerStarted","Data":"5dfe715d7ccad58493d43d4d54953a40525bfba8dc087bbe95a7d35c49548286"}
Oct 30 22:15:19.946422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.946404    5510 scope.go:122] "RemoveContainer" containerID="bf28e1cc0156990fe171b728b9dc6b756c0bb8ca7a2d96c0fdfb99b98042933b"
Oct 30 22:15:19.959576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.959553    5510 scope.go:122] "RemoveContainer" containerID="ea745707ff8b1565cd24cb9b8fddf0ab95b1f7f2f64a75e5a836e42fe2925c30"
Oct 30 22:15:19.973620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:19.973595    5510 scope.go:122] "RemoveContainer" containerID="f1516aabdf43ce1a1650c8219d22e65dfa3ef8df0f272fc0b2851d673a813c7c"
Oct 30 22:15:20.051164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.051130    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:20.178987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.178958    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:20.216093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.216067    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\" (UniqueName: \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\") pod \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") "
Oct 30 22:15:20.216288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.216274    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\") pod \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\" (UID: \"fa3fbdb7-1b93-4a94-b372-60523f6959a4\") "
Oct 30 22:15:20.216646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.216620    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca" pod "fa3fbdb7-1b93-4a94-b372-60523f6959a4" (UID: "fa3fbdb7-1b93-4a94-b372-60523f6959a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.217005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.216985    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa3fbdb7-1b93-4a94-b372-60523f6959a4-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.218576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.218552    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb" pod "fa3fbdb7-1b93-4a94-b372-60523f6959a4" (UID: "fa3fbdb7-1b93-4a94-b372-60523f6959a4"). InnerVolumeSpecName "kube-api-access-4vjzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.253977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.253949    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:20.318072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.318045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\") pod \"1e35b445-5371-4bbe-8d1c-a372b41daac2\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") "
Oct 30 22:15:20.318200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.318169    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\" (UniqueName: \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\") pod \"1e35b445-5371-4bbe-8d1c-a372b41daac2\" (UID: \"1e35b445-5371-4bbe-8d1c-a372b41daac2\") "
Oct 30 22:15:20.318387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.318366    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca" pod "1e35b445-5371-4bbe-8d1c-a372b41daac2" (UID: "1e35b445-5371-4bbe-8d1c-a372b41daac2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.318737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.318715    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjzb\" (UniqueName: \"kubernetes.io/projected/fa3fbdb7-1b93-4a94-b372-60523f6959a4-kube-api-access-4vjzb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.318737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.318735    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e35b445-5371-4bbe-8d1c-a372b41daac2-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.320068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.320047    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q" pod "1e35b445-5371-4bbe-8d1c-a372b41daac2" (UID: "1e35b445-5371-4bbe-8d1c-a372b41daac2"). InnerVolumeSpecName "kube-api-access-ldk2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.345091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.345070    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:20.378311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.378282    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:20.382005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.381972    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-17d809-29889975-szqr2"]
Oct 30 22:15:20.419340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.419313    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\") pod \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") "
Oct 30 22:15:20.419466 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.419405    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\" (UniqueName: \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\") pod \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\" (UID: \"3f19f275-fbb6-4ee5-ad16-71eab77cddd4\") "
Oct 30 22:15:20.419665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.419644    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca" pod "3f19f275-fbb6-4ee5-ad16-71eab77cddd4" (UID: "3f19f275-fbb6-4ee5-ad16-71eab77cddd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.419949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.419923    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldk2q\" (UniqueName: \"kubernetes.io/projected/1e35b445-5371-4bbe-8d1c-a372b41daac2-kube-api-access-ldk2q\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.419949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.419942    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.421408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.421388    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq" pod "3f19f275-fbb6-4ee5-ad16-71eab77cddd4" (UID: "3f19f275-fbb6-4ee5-ad16-71eab77cddd4"). InnerVolumeSpecName "kube-api-access-mh8hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.521596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.521509    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8hq\" (UniqueName: \"kubernetes.io/projected/3f19f275-fbb6-4ee5-ad16-71eab77cddd4-kube-api-access-mh8hq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.591394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.591366    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:20.716623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.716586    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:20.969404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.969376    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77" event={"ID":"1e35b445-5371-4bbe-8d1c-a372b41daac2","Type":"ContainerDied","Data":"1f168c579f41ff77095f7ad75d846586b5b07e3879f12b5014fc5ef7dc5e33d1"}
Oct 30 22:15:20.971402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.971382    5510 scope.go:122] "RemoveContainer" containerID="18b97efdf7e672b8daeade2c2620fe9050d9b578b1e9aefe1c9bf9927bbdf5ef"
Oct 30 22:15:20.971926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.969914    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"
Oct 30 22:15:20.972685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.972658    5510 generic.go:356] "Generic (PLEG): container finished" podID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerID="5dfe715d7ccad58493d43d4d54953a40525bfba8dc087bbe95a7d35c49548286" exitCode=60
Oct 30 22:15:20.973132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.972929    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerDied","Data":"5dfe715d7ccad58493d43d4d54953a40525bfba8dc087bbe95a7d35c49548286"}
Oct 30 22:15:20.973541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.973462    5510 scope.go:122] "RemoveContainer" containerID="5dfe715d7ccad58493d43d4d54953a40525bfba8dc087bbe95a7d35c49548286"
Oct 30 22:15:20.973876 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:20.973849    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8_odh-ai-gateway-infra(8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b"
Oct 30 22:15:20.976520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.976493    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg" event={"ID":"fa3fbdb7-1b93-4a94-b372-60523f6959a4","Type":"ContainerDied","Data":"23df3f23a408af37723fde08f8e4ad9f463958512c14c2b8033375392186a5d1"}
Oct 30 22:15:20.976638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.976591    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"
Oct 30 22:15:20.982035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.982009    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg" event={"ID":"3f19f275-fbb6-4ee5-ad16-71eab77cddd4","Type":"ContainerDied","Data":"470c122ca789cdc6f28c7425df023ffa55d3717fba5472d263badd1ef3152ed2"}
Oct 30 22:15:20.982156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.982096    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"
Oct 30 22:15:20.988119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:20.988088    5510 scope.go:122] "RemoveContainer" containerID="a05856cec88ddac85dab8d4518ecc8315d8da33555e1ee6e8930006a51e34218"
Oct 30 22:15:21.010667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.005233    5510 scope.go:122] "RemoveContainer" containerID="dfae2c8187e502785f425c8de0fd8cf207f9365a5700737ef14e4ae9d0acbaba"
Oct 30 22:15:21.026860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.021338    5510 scope.go:122] "RemoveContainer" containerID="ce110a0d59f23e2ee64e8f25975efd2a8030ef53613d473e6e3cc22486c70450"
Oct 30 22:15:21.130126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.130095    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" path="/var/lib/kubelet/pods/b050e62c-6750-4153-bd02-1c66ac2dfcee/volumes"
Oct 30 22:15:21.169616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.169588    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:21.232215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.232185    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\" (UniqueName: \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\") pod \"89df3d06-8903-46e7-92ea-e278438fffbd\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") "
Oct 30 22:15:21.232614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.232587    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\") pod \"89df3d06-8903-46e7-92ea-e278438fffbd\" (UID: \"89df3d06-8903-46e7-92ea-e278438fffbd\") "
Oct 30 22:15:21.232962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.232926    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca" pod "89df3d06-8903-46e7-92ea-e278438fffbd" (UID: "89df3d06-8903-46e7-92ea-e278438fffbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.233278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.233259    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df3d06-8903-46e7-92ea-e278438fffbd-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.235079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.235052    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk" pod "89df3d06-8903-46e7-92ea-e278438fffbd" (UID: "89df3d06-8903-46e7-92ea-e278438fffbd"). InnerVolumeSpecName "kube-api-access-g4prk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.263998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.263972    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:21.279308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.279285    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:21.298490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.298451    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:21.334697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.334658    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\") pod \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") "
Oct 30 22:15:21.334933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.334915    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\" (UniqueName: \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\") pod \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\" (UID: \"8c2c9b14-68a1-4f21-a5de-f10fd8053e52\") "
Oct 30 22:15:21.335110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.335083    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca" pod "8c2c9b14-68a1-4f21-a5de-f10fd8053e52" (UID: "8c2c9b14-68a1-4f21-a5de-f10fd8053e52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.335526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.335505    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4prk\" (UniqueName: \"kubernetes.io/projected/89df3d06-8903-46e7-92ea-e278438fffbd-kube-api-access-g4prk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.335652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.335530    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.337201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.337177    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88" pod "8c2c9b14-68a1-4f21-a5de-f10fd8053e52" (UID: "8c2c9b14-68a1-4f21-a5de-f10fd8053e52"). InnerVolumeSpecName "kube-api-access-jmf88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.360892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.360830    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:21.363598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.363574    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5be0d1-29889975-mck77"]
Oct 30 22:15:21.436346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.436317    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\" (UniqueName: \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\") pod \"f4453024-52ad-45b5-9596-f6a9145f598b\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") "
Oct 30 22:15:21.436515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.436413    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\") pod \"f4453024-52ad-45b5-9596-f6a9145f598b\" (UID: \"f4453024-52ad-45b5-9596-f6a9145f598b\") "
Oct 30 22:15:21.436826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.436803    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca" pod "f4453024-52ad-45b5-9596-f6a9145f598b" (UID: "f4453024-52ad-45b5-9596-f6a9145f598b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.437030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.437015    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmf88\" (UniqueName: \"kubernetes.io/projected/8c2c9b14-68a1-4f21-a5de-f10fd8053e52-kube-api-access-jmf88\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.437083 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.437033    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4453024-52ad-45b5-9596-f6a9145f598b-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.438510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.438484    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c" pod "f4453024-52ad-45b5-9596-f6a9145f598b" (UID: "f4453024-52ad-45b5-9596-f6a9145f598b"). InnerVolumeSpecName "kube-api-access-t4g8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.539294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.539208    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4g8c\" (UniqueName: \"kubernetes.io/projected/f4453024-52ad-45b5-9596-f6a9145f598b-kube-api-access-t4g8c\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.547387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.547353    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:21.696633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.696582    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:21.700311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.700283    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f776cb-29889975-55dbg"]
Oct 30 22:15:21.750071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.750047    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:21.804365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.804292    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:21.814323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.814290    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-14af5c-29889975-46swg"]
Oct 30 22:15:21.995343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.995310    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"
Oct 30 22:15:21.995738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.995313    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7" event={"ID":"89df3d06-8903-46e7-92ea-e278438fffbd","Type":"ContainerDied","Data":"5a42172f215087353022fcc8cfb03f0e8d96bbc6bd2a9cc8667447b097199c64"}
Oct 30 22:15:21.995738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.995472    5510 scope.go:122] "RemoveContainer" containerID="3fb8fbf24ee22a2fce5f98d4e71aac24ce012637fe2f9919e2034442b157bcac"
Oct 30 22:15:21.997116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.997090    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"
Oct 30 22:15:21.997236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.997093    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k" event={"ID":"f4453024-52ad-45b5-9596-f6a9145f598b","Type":"ContainerDied","Data":"a865e5dc2f87a50d7ec75ebb5ef9a0397f2735122e652cca5545fbc2d11023b2"}
Oct 30 22:15:21.998762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:21.998741    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"
Oct 30 22:15:22.000643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.000185    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb" event={"ID":"8c2c9b14-68a1-4f21-a5de-f10fd8053e52","Type":"ContainerDied","Data":"8d79af6df35c291518b8efa6a4d4c611805eb5a4d9723bd6df9d9c2daf8c078b"}
Oct 30 22:15:22.009492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.009289    5510 scope.go:122] "RemoveContainer" containerID="82e37fe69802a1c19a9e02b31f380a53e18a4d889c8073bdc4c73c558e2f7a07"
Oct 30 22:15:22.022091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.022069    5510 scope.go:122] "RemoveContainer" containerID="6a72380adfadca9d0d2202dc5ea2832b33845658ba48fe4baf08aa3b32d9172b"
Oct 30 22:15:22.153979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.153946    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:22.250095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.250067    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\") pod \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") "
Oct 30 22:15:22.250254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.250230    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\" (UniqueName: \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\") pod \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\" (UID: \"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b\") "
Oct 30 22:15:22.250483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.250463    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca" pod "8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" (UID: "8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:22.250990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.250966    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.252316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.252294    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt" pod "8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" (UID: "8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b"). InnerVolumeSpecName "kube-api-access-xplbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.352877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.352795    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xplbt\" (UniqueName: \"kubernetes.io/projected/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b-kube-api-access-xplbt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.529196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.529164    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:22.534702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.534675    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-c9dc10-29889975-cd58k"]
Oct 30 22:15:22.636356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.636259    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:22.640269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.640243    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4215ed-29889975-rf8qb"]
Oct 30 22:15:22.742714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.742677    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:22.748486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.748452    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2d1f91-29889975-4f9f7"]
Oct 30 22:15:22.952713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.952626    5510 scope.go:122] "RemoveContainer" containerID="cb7447d3ca757c29eb6dc4078d7c789ca9c7eb34bc0cf889b9742c2a943154af"
Oct 30 22:15:22.965084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:22.965058    5510 scope.go:122] "RemoveContainer" containerID="21ab180ccffafb67d81fd76ffc9805c9e52e2b620521dbb6dd8ee7a101acfbc8"
Oct 30 22:15:23.019265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.019234    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8" event={"ID":"8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b","Type":"ContainerDied","Data":"7afdd8556d649ed25339cdd0957cc489597a809eda957f23a58fd4db1a5c43f6"}
Oct 30 22:15:23.019687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.019288    5510 scope.go:122] "RemoveContainer" containerID="5dfe715d7ccad58493d43d4d54953a40525bfba8dc087bbe95a7d35c49548286"
Oct 30 22:15:23.019687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.019246    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"
Oct 30 22:15:23.128354 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.128322    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" path="/var/lib/kubelet/pods/1e35b445-5371-4bbe-8d1c-a372b41daac2/volumes"
Oct 30 22:15:23.128776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.128762    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" path="/var/lib/kubelet/pods/3f19f275-fbb6-4ee5-ad16-71eab77cddd4/volumes"
Oct 30 22:15:23.129197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.129183    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" path="/var/lib/kubelet/pods/89df3d06-8903-46e7-92ea-e278438fffbd/volumes"
Oct 30 22:15:23.129891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.129875    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" path="/var/lib/kubelet/pods/8c2c9b14-68a1-4f21-a5de-f10fd8053e52/volumes"
Oct 30 22:15:23.130311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.130296    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" path="/var/lib/kubelet/pods/f4453024-52ad-45b5-9596-f6a9145f598b/volumes"
Oct 30 22:15:23.130674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.130661    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" path="/var/lib/kubelet/pods/fa3fbdb7-1b93-4a94-b372-60523f6959a4/volumes"
Oct 30 22:15:23.196931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.193150    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:23.196931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.194930    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:23.203695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:23.203617    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-446040-29889975-dj9b8"]
Oct 30 22:15:24.705474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:24.705440    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:24.902829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:24.902470    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:25.158685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:25.153099    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" path="/var/lib/kubelet/pods/8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b/volumes"
Oct 30 22:15:26.582816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:26.582537    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.330803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:27.328802    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.497187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:27.497149    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.898982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:27.898935    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:28.081932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:28.081898    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:28.274909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:28.274872    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:30.387239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:30.387206    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:15:30.710493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:30.710461    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:30.880803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:30.878261    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:31.305225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:31.305198    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:31.489201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:31.489159    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:31.656852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:31.656724    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:32.915015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:32.914983    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:35.388371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:35.388299    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing" containerID="cri-o://b74a057101c1b66c867c37227ef59dc68cec25e61e659697f6a0933e2a6a2bf8" gracePeriod=25
Oct 30 22:15:37.820652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:37.820616    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:38.491370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:38.491332    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:38.694802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:38.684991    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:42.158906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:42.158865    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 22:15:42.159513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:42.159058    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" containerID="cri-o://1105d08b382fd1dafa834dcfab09673325b0c624c6c642e6c57445df616a804c" gracePeriod=30
Oct 30 22:15:42.584366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:42.584328    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:43.282917 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:43.282882    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:15:43.936889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:43.936847    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:47.162719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.162675    5510 patch_prober.go:29] interesting pod/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:47.163163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.162769    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:47.284257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.284191    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:47.393818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.393512    5510 generic.go:356] "Generic (PLEG): container finished" podID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerID="1105d08b382fd1dafa834dcfab09673325b0c624c6c642e6c57445df616a804c" exitCode=0
Oct 30 22:15:47.393818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.393717    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" event={"ID":"188c5dac-d2c7-4465-89c0-6d77551d9234","Type":"ContainerDied","Data":"1105d08b382fd1dafa834dcfab09673325b0c624c6c642e6c57445df616a804c"}
Oct 30 22:15:47.526005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.525967    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 22:15:47.627721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.627682    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.627946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.627748    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.627946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.627872    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.627999    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:47.628079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628026    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:47.628079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628029    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628110    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:47.628254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628171    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628204    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628251    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628282    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628338    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\") pod \"188c5dac-d2c7-4465-89c0-6d77551d9234\" (UID: \"188c5dac-d2c7-4465-89c0-6d77551d9234\") "
Oct 30 22:15:47.628453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628406    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:47.628617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.628591    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:47.629056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.629032    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.629161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.629061    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.629161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.629075    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.629161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.629089    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.629161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.629103    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/188c5dac-d2c7-4465-89c0-6d77551d9234-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.631099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.631077    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:47.631099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.631090    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:47.631235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.631094    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:47.631235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.631130    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb" pod "188c5dac-d2c7-4465-89c0-6d77551d9234" (UID: "188c5dac-d2c7-4465-89c0-6d77551d9234"). InnerVolumeSpecName "kube-api-access-9xlwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:47.730633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.730590    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.730633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.730623    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.730633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.730632    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlwb\" (UniqueName: \"kubernetes.io/projected/188c5dac-d2c7-4465-89c0-6d77551d9234-kube-api-access-9xlwb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:47.730928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:47.730642    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/188c5dac-d2c7-4465-89c0-6d77551d9234-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:48.404727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.404696    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"
Oct 30 22:15:48.404727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.404718    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk" event={"ID":"188c5dac-d2c7-4465-89c0-6d77551d9234","Type":"ContainerDied","Data":"9133a79dd53e729ef3637a1b292769587d6d36a95bbe9119acccc64caa8320de"}
Oct 30 22:15:48.405262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.404798    5510 scope.go:122] "RemoveContainer" containerID="1105d08b382fd1dafa834dcfab09673325b0c624c6c642e6c57445df616a804c"
Oct 30 22:15:48.606482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.606446    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 22:15:48.633015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.632980    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.652557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.652533    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-h6srk"]
Oct 30 22:15:48.786967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.786938    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:15:48.890792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:48.890734    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:15:49.051247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.051170    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.128701 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.128666    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" path="/var/lib/kubelet/pods/188c5dac-d2c7-4465-89c0-6d77551d9234/volumes"
Oct 30 22:15:49.244575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.244547    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.481803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.466133    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.541277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.541241    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:15:49.698314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:49.698180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:50.917681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:50.916018    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:51.096816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:51.096770    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:51.979801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:51.979355    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.788130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:53.788079    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" podUID="f30ba86c-53ca-4897-8ddc-738820b838d7" containerName="payload-processing" containerID="cri-o://6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200" gracePeriod=25
Oct 30 22:15:53.891749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:53.891709    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" podUID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" containerName="payload-pre-processing" containerID="cri-o://40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32" gracePeriod=25
Oct 30 22:15:53.938363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:53.938330    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:54.135238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.135046    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:15:54.240348 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.240314    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\") pod \"f30ba86c-53ca-4897-8ddc-738820b838d7\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") "
Oct 30 22:15:54.240542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.240423    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\" (UniqueName: \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\") pod \"f30ba86c-53ca-4897-8ddc-738820b838d7\" (UID: \"f30ba86c-53ca-4897-8ddc-738820b838d7\") "
Oct 30 22:15:54.240687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.240663    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume" pod "f30ba86c-53ca-4897-8ddc-738820b838d7" (UID: "f30ba86c-53ca-4897-8ddc-738820b838d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:54.241102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.241081    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ba86c-53ca-4897-8ddc-738820b838d7-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.242651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.242621    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft" pod "f30ba86c-53ca-4897-8ddc-738820b838d7" (UID: "f30ba86c-53ca-4897-8ddc-738820b838d7"). InnerVolumeSpecName "kube-api-access-6blft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:54.301454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.301428    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:15:54.341695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.341664    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\") pod \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") "
Oct 30 22:15:54.341864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.341708    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\" (UniqueName: \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\") pod \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\" (UID: \"131f3d9b-c66c-4fb0-89e0-aad437eb1768\") "
Oct 30 22:15:54.342069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.342043    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume" pod "131f3d9b-c66c-4fb0-89e0-aad437eb1768" (UID: "131f3d9b-c66c-4fb0-89e0-aad437eb1768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:54.342408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.342392    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131f3d9b-c66c-4fb0-89e0-aad437eb1768-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.342454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.342414    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blft\" (UniqueName: \"kubernetes.io/projected/f30ba86c-53ca-4897-8ddc-738820b838d7-kube-api-access-6blft\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.343861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.343839    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r" pod "131f3d9b-c66c-4fb0-89e0-aad437eb1768" (UID: "131f3d9b-c66c-4fb0-89e0-aad437eb1768"). InnerVolumeSpecName "kube-api-access-nvq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:54.444028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.443939    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvq5r\" (UniqueName: \"kubernetes.io/projected/131f3d9b-c66c-4fb0-89e0-aad437eb1768-kube-api-access-nvq5r\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.593028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.592988    5510 generic.go:356] "Generic (PLEG): container finished" podID="f30ba86c-53ca-4897-8ddc-738820b838d7" containerID="6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200" exitCode=0
Oct 30 22:15:54.593223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.593029    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"
Oct 30 22:15:54.593223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.593063    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" event={"ID":"f30ba86c-53ca-4897-8ddc-738820b838d7","Type":"ContainerDied","Data":"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"}
Oct 30 22:15:54.593223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.593120    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25" event={"ID":"f30ba86c-53ca-4897-8ddc-738820b838d7","Type":"ContainerDied","Data":"96495b7b0b459942c6c37f7534967c71a6182710a225903b0192860aa477488b"}
Oct 30 22:15:54.593223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.593150    5510 scope.go:122] "RemoveContainer" containerID="6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"
Oct 30 22:15:54.595111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.595087    5510 generic.go:356] "Generic (PLEG): container finished" podID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" containerID="40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32" exitCode=0
Oct 30 22:15:54.595257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.595241    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" event={"ID":"131f3d9b-c66c-4fb0-89e0-aad437eb1768","Type":"ContainerDied","Data":"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"}
Oct 30 22:15:54.595311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.595286    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk" event={"ID":"131f3d9b-c66c-4fb0-89e0-aad437eb1768","Type":"ContainerDied","Data":"8a39656ad193ae7e998e0077e6bc7d19a1202552144679e0a81cd80d9b640a00"}
Oct 30 22:15:54.595365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.595352    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"
Oct 30 22:15:54.607311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.607294    5510 scope.go:122] "RemoveContainer" containerID="6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"
Oct 30 22:15:54.607690 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:54.607666    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200\": container with ID starting with 6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200 not found: ID does not exist" containerID="6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"
Oct 30 22:15:54.607745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.607691    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200"} err="failed to get container status \"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200\": rpc error: code = NotFound desc = could not find container \"6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200\": container with ID starting with 6e643439162e58af140cdbff7c8f3dca01f7cbe6305aa29cedfe059766e61200 not found: ID does not exist"
Oct 30 22:15:54.607745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.607710    5510 scope.go:122] "RemoveContainer" containerID="40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"
Oct 30 22:15:54.617668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.617646    5510 scope.go:122] "RemoveContainer" containerID="40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"
Oct 30 22:15:54.617989 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:15:54.617963    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32\": container with ID starting with 40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32 not found: ID does not exist" containerID="40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"
Oct 30 22:15:54.618102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.618085    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32"} err="failed to get container status \"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32\": rpc error: code = NotFound desc = could not find container \"40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32\": container with ID starting with 40aa33829172b88534d50aa525bde4d7740fcd264172afc9b9dec0164a569c32 not found: ID does not exist"
Oct 30 22:15:54.746611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.746581    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:15:54.769802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.762041    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-446040-7854675894-fnj25"]
Oct 30 22:15:54.879099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.879017    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:15:54.882441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:54.882411    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-446040-77fb76fcf-tnmkk"]
Oct 30 22:15:55.130182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.130101    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" path="/var/lib/kubelet/pods/131f3d9b-c66c-4fb0-89e0-aad437eb1768/volumes"
Oct 30 22:15:55.131185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.131161    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30ba86c-53ca-4897-8ddc-738820b838d7" path="/var/lib/kubelet/pods/f30ba86c-53ca-4897-8ddc-738820b838d7/volumes"
Oct 30 22:15:55.616435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.616263    5510 generic.go:356] "Generic (PLEG): container finished" podID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerID="b74a057101c1b66c867c37227ef59dc68cec25e61e659697f6a0933e2a6a2bf8" exitCode=0
Oct 30 22:15:55.616435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.616322    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" event={"ID":"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2","Type":"ContainerDied","Data":"b74a057101c1b66c867c37227ef59dc68cec25e61e659697f6a0933e2a6a2bf8"}
Oct 30 22:15:55.721946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.721922    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:15:55.888934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.888889    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\") pod \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") "
Oct 30 22:15:55.889444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.889051    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\" (UniqueName: \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\") pod \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\" (UID: \"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2\") "
Oct 30 22:15:55.889444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.889285    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume" pod "188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" (UID: "188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:55.889765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.889738    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:55.891283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.891255    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r" pod "188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" (UID: "188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2"). InnerVolumeSpecName "kube-api-access-cdw9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:55.990863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:55.990825    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdw9r\" (UniqueName: \"kubernetes.io/projected/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2-kube-api-access-cdw9r\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:56.341460 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.341368    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:56.626543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.626463    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"
Oct 30 22:15:56.626543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.626501    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx" event={"ID":"188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2","Type":"ContainerDied","Data":"7b459f5fffbba92138bc45e64af827eee67fb4bdc7ed8d72d299fb7a9ae8118b"}
Oct 30 22:15:56.626808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.626556    5510 scope.go:122] "RemoveContainer" containerID="b74a057101c1b66c867c37227ef59dc68cec25e61e659697f6a0933e2a6a2bf8"
Oct 30 22:15:56.783238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.783204    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:15:56.827114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:56.827078    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-778d39d3-544ffbcf87-zxdpx"]
Oct 30 22:15:57.127724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:57.127690    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" path="/var/lib/kubelet/pods/188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2/volumes"
Oct 30 22:15:58.341913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:58.341870    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:15:58.342408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:15:58.341922    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:00.658194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.658108    5510 generic.go:356] "Generic (PLEG): container finished" podID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerID="6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4" exitCode=0
Oct 30 22:16:00.658716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.658183    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerDied","Data":"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"}
Oct 30 22:16:00.658716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.658648    5510 scope.go:122] "RemoveContainer" containerID="6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"
Oct 30 22:16:00.660599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.660570    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:00.660706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.660646    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:00.660706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.660681    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:16:00.831597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:00.831558    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:01.698118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:01.698075    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerStarted","Data":"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"}
Oct 30 22:16:02.609807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:02.609763    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:16:02.609999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:02.609941    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 22:16:02.610049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:02.609992    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 22:16:03.610508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:03.610468    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 22:16:03.610955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:03.610543    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 22:16:04.610829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:04.610499    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.610829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:04.610578    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 22:16:05.331726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:05.331693    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:05.610433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:05.610348    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.610433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:05.610415    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 22:16:06.610329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:06.610287    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.610861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:06.610362    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 22:16:07.614889 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:07.614854    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:16:07.615344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:07.615323    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:16:07.615986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:07.615966    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:16:09.897087 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:09.897050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.281096 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:16:13.281062    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:16:14.355347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:14.355314    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:19.127022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:19.126986    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:23.927903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:23.927868    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.250721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.250669    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.251543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.251519    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:16:28.349526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.349498    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:16:28.425316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.425280    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.600028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.599944    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.754861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.754829    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.907081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:28.907000    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:30.959452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:30.959416    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:31.113342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:31.113300    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.706054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:32.706019    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:33.251801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.251734    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" podUID="d8005249-6108-446f-a8f5-5e3fe342e034" containerName="payload-processing" containerID="cri-o://086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2" gracePeriod=25
Oct 30 22:16:33.350327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.350261    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" podUID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" containerName="payload-pre-processing" containerID="cri-o://680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864" gracePeriod=25
Oct 30 22:16:33.743543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.743516    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:16:33.842518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.842490    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:16:33.912454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.912376    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\" (UniqueName: \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\") pod \"d8005249-6108-446f-a8f5-5e3fe342e034\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") "
Oct 30 22:16:33.912656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.912644    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\") pod \"d8005249-6108-446f-a8f5-5e3fe342e034\" (UID: \"d8005249-6108-446f-a8f5-5e3fe342e034\") "
Oct 30 22:16:33.913093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.913068    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume" pod "d8005249-6108-446f-a8f5-5e3fe342e034" (UID: "d8005249-6108-446f-a8f5-5e3fe342e034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:33.914486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:33.914462    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx" pod "d8005249-6108-446f-a8f5-5e3fe342e034" (UID: "d8005249-6108-446f-a8f5-5e3fe342e034"). InnerVolumeSpecName "kube-api-access-nhxmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:34.014697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.014659    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\" (UniqueName: \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\") pod \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") "
Oct 30 22:16:34.014917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.014711    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\") pod \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\" (UID: \"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9\") "
Oct 30 22:16:34.015167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.015141    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume" pod "d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" (UID: "d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:34.015516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.015492    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8005249-6108-446f-a8f5-5e3fe342e034-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:34.015565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.015523    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxmx\" (UniqueName: \"kubernetes.io/projected/d8005249-6108-446f-a8f5-5e3fe342e034-kube-api-access-nhxmx\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:34.015565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.015540    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:34.017029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.017006    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p" pod "d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" (UID: "d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9"). InnerVolumeSpecName "kube-api-access-h6f4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:34.046254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.046219    5510 generic.go:356] "Generic (PLEG): container finished" podID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" containerID="680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864" exitCode=0
Oct 30 22:16:34.046427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.046273    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"
Oct 30 22:16:34.046427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.046298    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" event={"ID":"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9","Type":"ContainerDied","Data":"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"}
Oct 30 22:16:34.046427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.046373    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj" event={"ID":"d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9","Type":"ContainerDied","Data":"c8f58018a2ea7d8528f192b04de42079c8f6a447b9bce69eea08d047ed12aed8"}
Oct 30 22:16:34.046427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.046386    5510 scope.go:122] "RemoveContainer" containerID="680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"
Oct 30 22:16:34.047966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.047946    5510 generic.go:356] "Generic (PLEG): container finished" podID="d8005249-6108-446f-a8f5-5e3fe342e034" containerID="086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2" exitCode=0
Oct 30 22:16:34.048100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.047986    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"
Oct 30 22:16:34.048100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.048002    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" event={"ID":"d8005249-6108-446f-a8f5-5e3fe342e034","Type":"ContainerDied","Data":"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"}
Oct 30 22:16:34.048100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.048046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x" event={"ID":"d8005249-6108-446f-a8f5-5e3fe342e034","Type":"ContainerDied","Data":"59e2055cb0c25911adf1320a816e55a4360f72558c9486435984fa57ef477d33"}
Oct 30 22:16:34.058273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.057922    5510 scope.go:122] "RemoveContainer" containerID="680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"
Oct 30 22:16:34.058273 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:16:34.058231    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864\": container with ID starting with 680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864 not found: ID does not exist" containerID="680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"
Oct 30 22:16:34.058273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.058260    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864"} err="failed to get container status \"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864\": rpc error: code = NotFound desc = could not find container \"680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864\": container with ID starting with 680a47e9766882f396b556ecf866de5ad8e593fe8e0f486fd4128e82fe586864 not found: ID does not exist"
Oct 30 22:16:34.058551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.058276    5510 scope.go:122] "RemoveContainer" containerID="086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"
Oct 30 22:16:34.075130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.075106    5510 scope.go:122] "RemoveContainer" containerID="086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"
Oct 30 22:16:34.075424 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:16:34.075402    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2\": container with ID starting with 086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2 not found: ID does not exist" containerID="086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"
Oct 30 22:16:34.075500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.075432    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2"} err="failed to get container status \"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2\": rpc error: code = NotFound desc = could not find container \"086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2\": container with ID starting with 086ca4531949b7e1ccf2399eb4f331edc8e3a5490835775d49141ee26f377ab2 not found: ID does not exist"
Oct 30 22:16:34.116497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.116474    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6f4p\" (UniqueName: \"kubernetes.io/projected/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9-kube-api-access-h6f4p\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:34.203022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.202943    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:16:34.206429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.206401    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-6ea8da-6b6699d4cf-jcpcj"]
Oct 30 22:16:34.305665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.305638    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:16:34.314828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.314798    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-6ea8da-79c5494746-p5v9x"]
Oct 30 22:16:34.336829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.336799    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:34.745713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:34.745675    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:35.013240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:35.013165    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:35.126318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:35.126287    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8005249-6108-446f-a8f5-5e3fe342e034" path="/var/lib/kubelet/pods/d8005249-6108-446f-a8f5-5e3fe342e034/volumes"
Oct 30 22:16:35.126761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:35.126740    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" path="/var/lib/kubelet/pods/d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9/volumes"
Oct 30 22:16:35.677195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:35.677157    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.207322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:39.207286    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.687003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:42.686948    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:42.889910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:42.889875    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:43.020569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:43.020536    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:16:43.020847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:43.020818    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy" containerID="cri-o://8fffcecba968014675e1846c842c1c8f785d0c9e14417026ab00de9c63113931" gracePeriod=30
Oct 30 22:16:43.066087 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:43.066056    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:43.316595 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:16:43.316506    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:16:44.188468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:44.188427    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.301765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.301726    5510 generic.go:356] "Generic (PLEG): container finished" podID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerID="8fffcecba968014675e1846c842c1c8f785d0c9e14417026ab00de9c63113931" exitCode=0
Oct 30 22:16:48.302392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.301805    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerDied","Data":"8fffcecba968014675e1846c842c1c8f785d0c9e14417026ab00de9c63113931"}
Oct 30 22:16:48.302392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.301853    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl" event={"ID":"93cc0456-4000-4acf-bf92-6af5e5b87e65","Type":"ContainerDied","Data":"da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d"}
Oct 30 22:16:48.302392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.301871    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da989af82b8cb1b22badf8a7d0015af154d2c8e6ef7537db37e1bc3bb5b7fb5d"
Oct 30 22:16:48.302392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.301885    5510 scope.go:122] "RemoveContainer" containerID="929756a5e3b9cd713a388eac07a58f8f674f9175da24d0b581bcd994d83f021d"
Oct 30 22:16:48.317629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.317604    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:16:48.454975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.454944    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455060    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455111    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455147    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455298    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455297    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:48.455366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455358    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455395    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455427    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455473    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\") pod \"93cc0456-4000-4acf-bf92-6af5e5b87e65\" (UID: \"93cc0456-4000-4acf-bf92-6af5e5b87e65\") "
Oct 30 22:16:48.455672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.455571    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:48.456840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.456099    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.456840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.456125    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.456840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.456292    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:48.456840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.456489    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:48.456840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.456807    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:48.458700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.458675    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:48.459032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.458995    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:48.459703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.459677    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "kube-api-access-rcrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:48.460341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.460321    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo" pod "93cc0456-4000-4acf-bf92-6af5e5b87e65" (UID: "93cc0456-4000-4acf-bf92-6af5e5b87e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557360    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557391    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557400    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cc0456-4000-4acf-bf92-6af5e5b87e65-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557409    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557419    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrxm\" (UniqueName: \"kubernetes.io/projected/93cc0456-4000-4acf-bf92-6af5e5b87e65-kube-api-access-rcrxm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557429    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.557453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.557437    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cc0456-4000-4acf-bf92-6af5e5b87e65-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:48.827471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.827387    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:48.982827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:48.982773    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.154040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:49.154012    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.310522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:49.310487    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"
Oct 30 22:16:49.575593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:49.575556    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:16:49.578930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:49.578903    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-446040-openshift-default-77c7fc56b5-9bbsl"]
Oct 30 22:16:51.126788 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:51.126742    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" path="/var/lib/kubelet/pods/93cc0456-4000-4acf-bf92-6af5e5b87e65/volumes"
Oct 30 22:16:53.862334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:53.862297    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:58.758595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:16:58.758551    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.382546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:03.382507    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.571742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:03.571705    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.728951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:03.728921    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.622731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:05.622693    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:17:05.720662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:05.720623    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:17:05.990029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:05.989995    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:06.191215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:06.191183    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:07.739897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:07.739863    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:07.759724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:07.759690    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:17:09.529296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:09.529262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.623736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.623695    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" podUID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" containerName="payload-processing" containerID="cri-o://c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9" gracePeriod=25
Oct 30 22:17:10.721570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.721535    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" podUID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" containerName="payload-pre-processing" containerID="cri-o://2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3" gracePeriod=25
Oct 30 22:17:10.891192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.891129    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:17:10.916192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.916157    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\") pod \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") "
Oct 30 22:17:10.916447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.916425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\" (UniqueName: \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\") pod \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\" (UID: \"fef8a1d9-da05-4696-b7fb-15588f0c3efc\") "
Oct 30 22:17:10.917625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.917595    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume" pod "fef8a1d9-da05-4696-b7fb-15588f0c3efc" (UID: "fef8a1d9-da05-4696-b7fb-15588f0c3efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:10.921162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.921138    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22" pod "fef8a1d9-da05-4696-b7fb-15588f0c3efc" (UID: "fef8a1d9-da05-4696-b7fb-15588f0c3efc"). InnerVolumeSpecName "kube-api-access-57f22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:10.985266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:10.985243    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:17:11.017992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.017958    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\") pod \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") "
Oct 30 22:17:11.018161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.018017    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\" (UniqueName: \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\") pod \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\" (UID: \"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04\") "
Oct 30 22:17:11.018419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.018391    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume" pod "6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" (UID: "6a5d6e3f-57b0-49ac-bbaf-97a609b78a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:11.018540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.018522    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8a1d9-da05-4696-b7fb-15588f0c3efc-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:11.018671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.018545    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:11.018671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.018560    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57f22\" (UniqueName: \"kubernetes.io/projected/fef8a1d9-da05-4696-b7fb-15588f0c3efc-kube-api-access-57f22\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:11.020655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.020634    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n" pod "6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" (UID: "6a5d6e3f-57b0-49ac-bbaf-97a609b78a04"). InnerVolumeSpecName "kube-api-access-vl64n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:11.120392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.120354    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl64n\" (UniqueName: \"kubernetes.io/projected/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04-kube-api-access-vl64n\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:11.515453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.515414    5510 generic.go:356] "Generic (PLEG): container finished" podID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" containerID="c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9" exitCode=0
Oct 30 22:17:11.515634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.515478    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"
Oct 30 22:17:11.515634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.515488    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" event={"ID":"fef8a1d9-da05-4696-b7fb-15588f0c3efc","Type":"ContainerDied","Data":"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"}
Oct 30 22:17:11.515634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.515572    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw" event={"ID":"fef8a1d9-da05-4696-b7fb-15588f0c3efc","Type":"ContainerDied","Data":"41e8e50972425b4c2320bf4f85d75aa08085b4512cc99aa4056ce143abeaf77e"}
Oct 30 22:17:11.515634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.515597    5510 scope.go:122] "RemoveContainer" containerID="c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"
Oct 30 22:17:11.516943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.516923    5510 generic.go:356] "Generic (PLEG): container finished" podID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" containerID="2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3" exitCode=0
Oct 30 22:17:11.517074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.516949    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" event={"ID":"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04","Type":"ContainerDied","Data":"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"}
Oct 30 22:17:11.517074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.516988    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"
Oct 30 22:17:11.517074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.517003    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs" event={"ID":"6a5d6e3f-57b0-49ac-bbaf-97a609b78a04","Type":"ContainerDied","Data":"994f57712c96828606be345e7e5e998a4f35ef0596ad6d854556f2895a557d25"}
Oct 30 22:17:11.526001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.525982    5510 scope.go:122] "RemoveContainer" containerID="c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"
Oct 30 22:17:11.526278 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:11.526255    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9\": container with ID starting with c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9 not found: ID does not exist" containerID="c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"
Oct 30 22:17:11.526344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.526287    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9"} err="failed to get container status \"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9\": rpc error: code = NotFound desc = could not find container \"c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9\": container with ID starting with c237ed32664184564c0b78abae3c96fec95324e27f27e8bf8f6fc6de1e6aeff9 not found: ID does not exist"
Oct 30 22:17:11.526344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.526309    5510 scope.go:122] "RemoveContainer" containerID="2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"
Oct 30 22:17:11.536187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.536170    5510 scope.go:122] "RemoveContainer" containerID="2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"
Oct 30 22:17:11.536446 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:11.536422    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3\": container with ID starting with 2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3 not found: ID does not exist" containerID="2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"
Oct 30 22:17:11.536541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.536447    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3"} err="failed to get container status \"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3\": rpc error: code = NotFound desc = could not find container \"2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3\": container with ID starting with 2a1829aba7b82c30fbaad2c464d5f9d32aa27f6b856da5c40877231aeca053c3 not found: ID does not exist"
Oct 30 22:17:11.677151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.677108    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:17:11.722804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.722764    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-411ff1-7986d6747f-k4kjs"]
Oct 30 22:17:11.794371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.794290    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:17:11.797417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:11.797395    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-411ff1-75fdbd68db-ktscw"]
Oct 30 22:17:13.126389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:13.126358    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" path="/var/lib/kubelet/pods/6a5d6e3f-57b0-49ac-bbaf-97a609b78a04/volumes"
Oct 30 22:17:13.126908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:13.126772    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" path="/var/lib/kubelet/pods/fef8a1d9-da05-4696-b7fb-15588f0c3efc/volumes"
Oct 30 22:17:13.286448 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:13.286409    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:17:14.300310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:14.300277    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.649428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:14.649356    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.861565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:14.861312    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.834241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:18.834201    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.331942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:20.331901    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.495214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:20.495181    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.339485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:23.339453    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:27.118802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.118728    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:17:27.119317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.119034    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" podUID="118964a8-172f-4d71-a507-addbaca906ef" containerName="maas-api" containerID="cri-o://a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859" gracePeriod=30
Oct 30 22:17:27.224879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.224143    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:27.614556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.614523    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:17:27.642633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.642595    5510 generic.go:356] "Generic (PLEG): container finished" podID="118964a8-172f-4d71-a507-addbaca906ef" containerID="a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859" exitCode=0
Oct 30 22:17:27.642819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.642634    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"
Oct 30 22:17:27.642819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.642714    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" event={"ID":"118964a8-172f-4d71-a507-addbaca906ef","Type":"ContainerDied","Data":"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"}
Oct 30 22:17:27.642819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.642766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j" event={"ID":"118964a8-172f-4d71-a507-addbaca906ef","Type":"ContainerDied","Data":"30572e707e89f84362e21795ecdafcc03d9fb8ebd160bae54f8f700ba16a5b92"}
Oct 30 22:17:27.642819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.642814    5510 scope.go:122] "RemoveContainer" containerID="a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"
Oct 30 22:17:27.654326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.654306    5510 scope.go:122] "RemoveContainer" containerID="a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"
Oct 30 22:17:27.654630 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:27.654606    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859\": container with ID starting with a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859 not found: ID does not exist" containerID="a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"
Oct 30 22:17:27.654728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.654635    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859"} err="failed to get container status \"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859\": rpc error: code = NotFound desc = could not find container \"a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859\": container with ID starting with a53e0ce67529877f562e95bf1e20c648924a862d034f0dce28e90ebda7f75859 not found: ID does not exist"
Oct 30 22:17:27.700870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.700774    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\") pod \"118964a8-172f-4d71-a507-addbaca906ef\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") "
Oct 30 22:17:27.701015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.700870    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\" (UniqueName: \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\") pod \"118964a8-172f-4d71-a507-addbaca906ef\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") "
Oct 30 22:17:27.701089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.701021    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\") pod \"118964a8-172f-4d71-a507-addbaca906ef\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") "
Oct 30 22:17:27.701089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.701068    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") pod \"118964a8-172f-4d71-a507-addbaca906ef\" (UID: \"118964a8-172f-4d71-a507-addbaca906ef\") "
Oct 30 22:17:27.701406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.701372    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca" pod "118964a8-172f-4d71-a507-addbaca906ef" (UID: "118964a8-172f-4d71-a507-addbaca906ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:27.701571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.701554    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118964a8-172f-4d71-a507-addbaca906ef-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.702851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.702821    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs" pod "118964a8-172f-4d71-a507-addbaca906ef" (UID: "118964a8-172f-4d71-a507-addbaca906ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:27.702946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.702893    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff" pod "118964a8-172f-4d71-a507-addbaca906ef" (UID: "118964a8-172f-4d71-a507-addbaca906ef"). InnerVolumeSpecName "kube-api-access-z82ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:27.703004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.702990    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls" pod "118964a8-172f-4d71-a507-addbaca906ef" (UID: "118964a8-172f-4d71-a507-addbaca906ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:27.802952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.802924    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.802952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.802950    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118964a8-172f-4d71-a507-addbaca906ef-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:27.803136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:27.802960    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z82ff\" (UniqueName: \"kubernetes.io/projected/118964a8-172f-4d71-a507-addbaca906ef-kube-api-access-z82ff\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:28.077839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:28.077767    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:17:28.101304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:28.101266    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-6ea8da-85668cb997-4nh8j"]
Oct 30 22:17:28.560416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:28.560385    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.711110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:28.711079    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.861089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:28.861009    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:29.132438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:29.132361    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118964a8-172f-4d71-a507-addbaca906ef" path="/var/lib/kubelet/pods/118964a8-172f-4d71-a507-addbaca906ef/volumes"
Oct 30 22:17:29.480938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:29.480909    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:33.496994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:33.496956    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:35.496364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:35.496328    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:35.644000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:35.643965    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:35.834049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:35.833959    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:17:35.834204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:35.834182    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" containerID="cri-o://5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca" gracePeriod=30
Oct 30 22:17:35.851414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:35.851388    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:36.755033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:36.754994    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:36.755505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:36.755056    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:38.371565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:38.371519    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:41.111490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.111431    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:17:41.219594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219563    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219663    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219705    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219734    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219766    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219856    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.219936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219893    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.220103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219944    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.220103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219947    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:41.220103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.219981    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:41.220103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220041    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:41.220103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220055    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:41.220385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220162    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:41.220385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220188    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\") pod \"10f360d1-daf8-45e0-9c66-71327d4f5b28\" (UID: \"10f360d1-daf8-45e0-9c66-71327d4f5b28\") "
Oct 30 22:17:41.220934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220908    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.221074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220936    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.221074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220950    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.221074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220967    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.221074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.220979    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f360d1-daf8-45e0-9c66-71327d4f5b28-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.222197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.222166    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:41.222297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.222218    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "kube-api-access-jb7wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:41.222438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.222423    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:41.222490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.222469    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token" pod "10f360d1-daf8-45e0-9c66-71327d4f5b28" (UID: "10f360d1-daf8-45e0-9c66-71327d4f5b28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:41.321961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.321929    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.321961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.321957    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb7wh\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-kube-api-access-jb7wh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.322174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.321979    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.322174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.321993    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f360d1-daf8-45e0-9c66-71327d4f5b28-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.745637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.745601    5510 generic.go:356] "Generic (PLEG): container finished" podID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerID="5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca" exitCode=0
Oct 30 22:17:41.745832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.745665    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"
Oct 30 22:17:41.745832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.745699    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerDied","Data":"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"}
Oct 30 22:17:41.745832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.745768    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl" event={"ID":"10f360d1-daf8-45e0-9c66-71327d4f5b28","Type":"ContainerDied","Data":"18b05f190cf8bc5db99aaa258e0bd96a52330bf6c74962a55f38fe100d26aaf4"}
Oct 30 22:17:41.745832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.745810    5510 scope.go:122] "RemoveContainer" containerID="5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"
Oct 30 22:17:41.758020 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.758005    5510 scope.go:122] "RemoveContainer" containerID="dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"
Oct 30 22:17:41.767668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.767651    5510 scope.go:122] "RemoveContainer" containerID="5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"
Oct 30 22:17:41.767963 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:41.767943    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca\": container with ID starting with 5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca not found: ID does not exist" containerID="5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"
Oct 30 22:17:41.768051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.767971    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca"} err="failed to get container status \"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca\": rpc error: code = NotFound desc = could not find container \"5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca\": container with ID starting with 5c68bff800e51aca630a00e79814ad337199cc62a85a7baa130a69b4088887ca not found: ID does not exist"
Oct 30 22:17:41.768051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.767989    5510 scope.go:122] "RemoveContainer" containerID="dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"
Oct 30 22:17:41.768480 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:41.768461    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50\": container with ID starting with dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50 not found: ID does not exist" containerID="dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"
Oct 30 22:17:41.768531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.768485    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50"} err="failed to get container status \"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50\": rpc error: code = NotFound desc = could not find container \"dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50\": container with ID starting with dea7dd83736e29b63f5d8462d7542b4487744d08165d258525cdeaa0cd5f7f50 not found: ID does not exist"
Oct 30 22:17:41.900085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.900047    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:17:41.905269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:41.905242    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-6ea8da-openshift-default-664d7c855c-8j7dl"]
Oct 30 22:17:42.243220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.243185    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:17:42.243625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.243369    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" podUID="f3efe962-3a6f-4529-b28c-c4f363de96e3" containerName="maas-api" containerID="cri-o://b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de" gracePeriod=30
Oct 30 22:17:42.357833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.357772    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:42.631636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.631604    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:17:42.763974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.763924    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\") pod \"f3efe962-3a6f-4529-b28c-c4f363de96e3\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") "
Oct 30 22:17:42.764261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.764239    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\") pod \"f3efe962-3a6f-4529-b28c-c4f363de96e3\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") "
Oct 30 22:17:42.764533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.764517    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\" (UniqueName: \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\") pod \"f3efe962-3a6f-4529-b28c-c4f363de96e3\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") "
Oct 30 22:17:42.764732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.764715    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") pod \"f3efe962-3a6f-4529-b28c-c4f363de96e3\" (UID: \"f3efe962-3a6f-4529-b28c-c4f363de96e3\") "
Oct 30 22:17:42.766716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.766674    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca" pod "f3efe962-3a6f-4529-b28c-c4f363de96e3" (UID: "f3efe962-3a6f-4529-b28c-c4f363de96e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:42.775933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.775885    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls" pod "f3efe962-3a6f-4529-b28c-c4f363de96e3" (UID: "f3efe962-3a6f-4529-b28c-c4f363de96e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:42.779461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.779437    5510 generic.go:356] "Generic (PLEG): container finished" podID="f3efe962-3a6f-4529-b28c-c4f363de96e3" containerID="b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de" exitCode=0
Oct 30 22:17:42.779672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.779657    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" event={"ID":"f3efe962-3a6f-4529-b28c-c4f363de96e3","Type":"ContainerDied","Data":"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"}
Oct 30 22:17:42.779868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.779844    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"
Oct 30 22:17:42.780540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.780507    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs" pod "f3efe962-3a6f-4529-b28c-c4f363de96e3" (UID: "f3efe962-3a6f-4529-b28c-c4f363de96e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:42.780659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.780523    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv" event={"ID":"f3efe962-3a6f-4529-b28c-c4f363de96e3","Type":"ContainerDied","Data":"c141d8bfe851cebd6fb9402c80cb3079cdebb57efa9cec3c8f88ba48f8f804be"}
Oct 30 22:17:42.780840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.780546    5510 scope.go:122] "RemoveContainer" containerID="b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"
Oct 30 22:17:42.782244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.782000    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x" pod "f3efe962-3a6f-4529-b28c-c4f363de96e3" (UID: "f3efe962-3a6f-4529-b28c-c4f363de96e3"). InnerVolumeSpecName "kube-api-access-lnn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:42.805824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.805752    5510 scope.go:122] "RemoveContainer" containerID="b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"
Oct 30 22:17:42.806146 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:42.806126    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de\": container with ID starting with b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de not found: ID does not exist" containerID="b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"
Oct 30 22:17:42.806217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.806154    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de"} err="failed to get container status \"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de\": rpc error: code = NotFound desc = could not find container \"b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de\": container with ID starting with b895548f603e1a7dcefb621482c5eb8c523562a47df5a74f91e94cce822062de not found: ID does not exist"
Oct 30 22:17:42.867362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.867285    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnn4x\" (UniqueName: \"kubernetes.io/projected/f3efe962-3a6f-4529-b28c-c4f363de96e3-kube-api-access-lnn4x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.867362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.867317    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.867362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.867328    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3efe962-3a6f-4529-b28c-c4f363de96e3-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:42.867362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:42.867337    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3efe962-3a6f-4529-b28c-c4f363de96e3-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.181848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:43.171011    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" path="/var/lib/kubelet/pods/10f360d1-daf8-45e0-9c66-71327d4f5b28/volumes"
Oct 30 22:17:43.237866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:43.237822    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:17:43.242994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:43.242912    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25de17-f888876bc-crksv"]
Oct 30 22:17:43.286258 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:43.286220    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:17:43.756967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:43.756928    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:43.977503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:43.977464    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:44.203256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:44.203183    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:44.669279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:44.669243    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.127074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:45.127042    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3efe962-3a6f-4529-b28c-c4f363de96e3" path="/var/lib/kubelet/pods/f3efe962-3a6f-4529-b28c-c4f363de96e3/volumes"
Oct 30 22:17:48.059995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:48.059963    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:17:48.439534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:48.439453    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.609044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:48.609014    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:48.793708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:48.793667    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.720058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:50.719983    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.862390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:50.862353    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:51.041891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:51.041808    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.060739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.060699    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" podUID="0920ec87-5acb-4457-80b5-1c8479e06c07" containerName="payload-processing" containerID="cri-o://09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29" gracePeriod=25
Oct 30 22:17:53.169226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.169198    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.338893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.338869    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:17:53.494001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.491012    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\") pod \"0920ec87-5acb-4457-80b5-1c8479e06c07\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") "
Oct 30 22:17:53.494001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.491128    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\" (UniqueName: \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\") pod \"0920ec87-5acb-4457-80b5-1c8479e06c07\" (UID: \"0920ec87-5acb-4457-80b5-1c8479e06c07\") "
Oct 30 22:17:53.494001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.492465    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume" pod "0920ec87-5acb-4457-80b5-1c8479e06c07" (UID: "0920ec87-5acb-4457-80b5-1c8479e06c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:53.498803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.495609    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx" pod "0920ec87-5acb-4457-80b5-1c8479e06c07" (UID: "0920ec87-5acb-4457-80b5-1c8479e06c07"). InnerVolumeSpecName "kube-api-access-4hfhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:53.593489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.593414    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0920ec87-5acb-4457-80b5-1c8479e06c07-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.593686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.593651    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hfhx\" (UniqueName: \"kubernetes.io/projected/0920ec87-5acb-4457-80b5-1c8479e06c07-kube-api-access-4hfhx\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.662059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.662020    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:17:53.663039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663016    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f3efe962-3a6f-4529-b28c-c4f363de96e3" containerName="maas-api"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663041    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3efe962-3a6f-4529-b28c-c4f363de96e3" containerName="maas-api"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663053    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663061    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663073    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663081    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663092    5510 cpu_manager.go:414] "containerMap: removing container" podUID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" containerName="payload-pre-processing"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663100    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" containerName="payload-pre-processing"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663111    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663118    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663133    5510 cpu_manager.go:414] "containerMap: removing container" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.663144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663140    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663150    5510 cpu_manager.go:414] "containerMap: removing container" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663159    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663174    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d8005249-6108-446f-a8f5-5e3fe342e034" containerName="payload-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663182    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8005249-6108-446f-a8f5-5e3fe342e034" containerName="payload-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663196    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" containerName="payload-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663203    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" containerName="payload-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663215    5510 cpu_manager.go:414] "containerMap: removing container" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663222    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663232    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663239    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663262    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663269    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663280    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663287    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663296    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663303    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663320    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663327    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663334    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663341    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663355    5510 cpu_manager.go:414] "containerMap: removing container" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663362    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663380    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663388    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663402    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663409    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" containerName="payload-pre-processing"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663418    5510 cpu_manager.go:414] "containerMap: removing container" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663426    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663439    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663446    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663457    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663464    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663474    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663480    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663494    5510 cpu_manager.go:414] "containerMap: removing container" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:17:53.663491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663502    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663517    5510 cpu_manager.go:414] "containerMap: removing container" podUID="118964a8-172f-4d71-a507-addbaca906ef" containerName="maas-api"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663524    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118964a8-172f-4d71-a507-addbaca906ef" containerName="maas-api"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663533    5510 cpu_manager.go:414] "containerMap: removing container" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663540    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663552    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f30ba86c-53ca-4897-8ddc-738820b838d7" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663560    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30ba86c-53ca-4897-8ddc-738820b838d7" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663570    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663577    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663595    5510 cpu_manager.go:414] "containerMap: removing container" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663601    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663613    5510 cpu_manager.go:414] "containerMap: removing container" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663620    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663631    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663638    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663654    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663661    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663674    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0920ec87-5acb-4457-80b5-1c8479e06c07" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663682    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0920ec87-5acb-4457-80b5-1c8479e06c07" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663699    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663706    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663878    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663893    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663905    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b0c94e-72fd-4d98-b5e3-22eff6fb36e9" containerName="payload-pre-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663917    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663926    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663935    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663942    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663953    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663963    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="118964a8-172f-4d71-a507-addbaca906ef" containerName="maas-api"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663972    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="188c5dac-d2c7-4465-89c0-6d77551d9234" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663981    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30ba86c-53ca-4897-8ddc-738820b838d7" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.663989    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664000    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664009    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664017    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664027    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664037    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0920ec87-5acb-4457-80b5-1c8479e06c07" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664045    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664054    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664066    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3efe962-3a6f-4529-b28c-c4f363de96e3" containerName="maas-api"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664077    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664086    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664093    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="131f3d9b-c66c-4fb0-89e0-aad437eb1768" containerName="payload-pre-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664103    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664112    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664122    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="188c5016-ab8f-49cd-8bd7-95ffb1bcf6a2" containerName="payload-pre-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664130    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8005249-6108-446f-a8f5-5e3fe342e034" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664138    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8a1d9-da05-4696-b7fb-15588f0c3efc" containerName="payload-processing"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664148    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f360d1-daf8-45e0-9c66-71327d4f5b28" containerName="istio-proxy"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664157    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664165    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664173    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:17:53.664582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.664183    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5d6e3f-57b0-49ac-bbaf-97a609b78a04" containerName="payload-pre-processing"
Oct 30 22:17:53.668419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.668400    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.691546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.691524    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.695100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695077    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695119    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695284    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695338    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695462    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695494    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695533    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvqw\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695659    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.695732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.695697    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.716954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.716925    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3aa85e-openshift-default-dockercfg-6nm6c\""
Oct 30 22:17:53.720095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.720067    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.798766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.798882    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.798995    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799029    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799070    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvqw\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799224    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799293    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799381    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.799456    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.800187    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.800949    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.801208    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.801730    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.817805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.802009    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.880861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.864409    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.880861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.874681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvqw\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.883640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.883560    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.907163    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.908811    5510 generic.go:356] "Generic (PLEG): container finished" podID="0920ec87-5acb-4457-80b5-1c8479e06c07" containerID="09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29" exitCode=0
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.909000    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" event={"ID":"0920ec87-5acb-4457-80b5-1c8479e06c07","Type":"ContainerDied","Data":"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"}
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.909051    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm" event={"ID":"0920ec87-5acb-4457-80b5-1c8479e06c07","Type":"ContainerDied","Data":"39f8193fabe02e2381acba74d232f2b3bfa84a3130f283f586b93a46a55c1bdc"}
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.909079    5510 scope.go:122] "RemoveContainer" containerID="09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.909228    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"
Oct 30 22:17:53.925376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.913088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\") pod \"e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") " pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:53.956115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.953183    5510 scope.go:122] "RemoveContainer" containerID="09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"
Oct 30 22:17:53.956115 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:17:53.953543    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29\": container with ID starting with 09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29 not found: ID does not exist" containerID="09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"
Oct 30 22:17:53.956115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.953580    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29"} err="failed to get container status \"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29\": rpc error: code = NotFound desc = could not find container \"09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29\": container with ID starting with 09d62a584822cee643e2803b749a42f065f34f8260758e42de273b559594af29 not found: ID does not exist"
Oct 30 22:17:53.987804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:53.986951    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:54.038022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.036257    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:54.098851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.097996    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:17:54.108180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.108147    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-c9dc10-cd748d646-6dxbm"]
Oct 30 22:17:54.470976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.470935    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:17:54.476737 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:17:54.476707    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083c59b3_fcea_4225_b6a9_e46fdd1f8b67.slice/crio-511d6e4f38b63e1d101512524da10018d05236f01932a0fc7a619d9d50c1e98f WatchSource:0}: Error finding container 511d6e4f38b63e1d101512524da10018d05236f01932a0fc7a619d9d50c1e98f: Status 404 returned error can't find the container with id 511d6e4f38b63e1d101512524da10018d05236f01932a0fc7a619d9d50c1e98f
Oct 30 22:17:54.480178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.480145    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:54.480305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.480231    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:54.480305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.480278    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:17:54.918527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.918439    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerStarted","Data":"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"}
Oct 30 22:17:54.918527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.918495    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerStarted","Data":"511d6e4f38b63e1d101512524da10018d05236f01932a0fc7a619d9d50c1e98f"}
Oct 30 22:17:54.988977 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:54.988940    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:55.028818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:55.028754    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podStartSLOduration=2.028740322 podStartE2EDuration="2.028740322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:55.027829947 +0000 UTC m=+2084.011736823" watchObservedRunningTime="2026-10-30 22:17:55.028740322 +0000 UTC m=+2084.012647197"
Oct 30 22:17:55.127559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:55.127526    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0920ec87-5acb-4457-80b5-1c8479e06c07" path="/var/lib/kubelet/pods/0920ec87-5acb-4457-80b5-1c8479e06c07/volumes"
Oct 30 22:17:56.098445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:56.098121    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:56.477805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:56.477301    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:17:56.933822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:56.933725    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:56.934816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:56.934765    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:17:58.564563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:17:58.564528    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.558045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:02.558005    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:18:02.558551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:02.558216    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" podUID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" containerName="maas-api" containerID="cri-o://a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6" gracePeriod=30
Oct 30 22:18:02.686440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:02.686406    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.870696 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:02.870666    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:18:03.024022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.023983    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\") pod \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") "
Oct 30 22:18:03.024227 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.024127    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\" (UniqueName: \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\") pod \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") "
Oct 30 22:18:03.024299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.024261    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") pod \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") "
Oct 30 22:18:03.024395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.024375    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\") pod \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\" (UID: \"8249f19f-1397-40c4-8b2c-2a8d75d6ff36\") "
Oct 30 22:18:03.028864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.028347    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca" pod "8249f19f-1397-40c4-8b2c-2a8d75d6ff36" (UID: "8249f19f-1397-40c4-8b2c-2a8d75d6ff36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:03.031748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.031045    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs" pod "8249f19f-1397-40c4-8b2c-2a8d75d6ff36" (UID: "8249f19f-1397-40c4-8b2c-2a8d75d6ff36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:03.031748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.031065    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls" pod "8249f19f-1397-40c4-8b2c-2a8d75d6ff36" (UID: "8249f19f-1397-40c4-8b2c-2a8d75d6ff36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:03.037422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.037369    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls" pod "8249f19f-1397-40c4-8b2c-2a8d75d6ff36" (UID: "8249f19f-1397-40c4-8b2c-2a8d75d6ff36"). InnerVolumeSpecName "kube-api-access-zh8ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:03.041861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.039526    5510 generic.go:356] "Generic (PLEG): container finished" podID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" containerID="a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6" exitCode=0
Oct 30 22:18:03.041861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.039600    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" event={"ID":"8249f19f-1397-40c4-8b2c-2a8d75d6ff36","Type":"ContainerDied","Data":"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"}
Oct 30 22:18:03.041861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.039637    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7" event={"ID":"8249f19f-1397-40c4-8b2c-2a8d75d6ff36","Type":"ContainerDied","Data":"61dfb39a6730aa84add3f84d4050d7602628e069ecb08e0bc9a3038485139f6d"}
Oct 30 22:18:03.041861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.039657    5510 scope.go:122] "RemoveContainer" containerID="a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"
Oct 30 22:18:03.041861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.039804    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"
Oct 30 22:18:03.080960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.080929    5510 scope.go:122] "RemoveContainer" containerID="a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"
Oct 30 22:18:03.081418 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:03.081394    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6\": container with ID starting with a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6 not found: ID does not exist" containerID="a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"
Oct 30 22:18:03.081517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.081429    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6"} err="failed to get container status \"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6\": rpc error: code = NotFound desc = could not find container \"a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6\": container with ID starting with a29caddfc9c0ab6a4462fd7190dffe2eb7d9d3eb15551478ac0268555dc066c6 not found: ID does not exist"
Oct 30 22:18:03.128598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.128515    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:03.128598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.128544    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8ls\" (UniqueName: \"kubernetes.io/projected/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-kube-api-access-zh8ls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:03.128598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.128560    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:03.128598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.128571    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8249f19f-1397-40c4-8b2c-2a8d75d6ff36-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:03.203167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.203129    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:18:03.206002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:03.205968    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-r6tk7"]
Oct 30 22:18:04.103424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:04.103389    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.300676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:04.300642    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:04.464026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:04.463995    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:05.126173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:05.126139    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" path="/var/lib/kubelet/pods/8249f19f-1397-40c4-8b2c-2a8d75d6ff36/volumes"
Oct 30 22:18:07.372395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:07.372359    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.944981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:07.944937    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:08.778195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:08.778153    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:09.172733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:09.172645    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.730810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:11.730767    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:18:12.236604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:12.236576    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:18:13.282446 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:13.282407    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:18:14.161547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:14.161519    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:17.866723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:17.866687    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.034036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:18.034000    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.200563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:18.200486    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:18.741269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:18.741234    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:19.085624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:19.085542    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:22.411686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:22.411656    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:22.568983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:22.568947    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:23.124256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:23.124222    5510 scope.go:122] "RemoveContainer" containerID="8fffcecba968014675e1846c842c1c8f785d0c9e14417026ab00de9c63113931"
Oct 30 22:18:23.259371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:23.259340    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:24.199799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.195823    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:24.400807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.399510    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:24.795802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.795500    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796543    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796569    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e35b445-5371-4bbe-8d1c-a372b41daac2" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796606    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796615    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796667    5510 cpu_manager.go:414] "containerMap: removing container" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796676    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796700    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" containerName="maas-api"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796708    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" containerName="maas-api"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796737    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796747    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f19f275-fbb6-4ee5-ad16-71eab77cddd4" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796765    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796775    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b050e62c-6750-4153-bd02-1c66ac2dfcee" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796965    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9b14-68a1-4f21-a5de-f10fd8053e52" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796985    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="89df3d06-8903-46e7-92ea-e278438fffbd" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.796998    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cc0456-4000-4acf-bf92-6af5e5b87e65" containerName="istio-proxy"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.797009    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3fbdb7-1b93-4a94-b372-60523f6959a4" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.797021    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb1ed4e-d963-4f3b-ac0c-5d6e8c2f334b" containerName="cleanup"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.797032    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8249f19f-1397-40c4-8b2c-2a8d75d6ff36" containerName="maas-api"
Oct 30 22:18:24.798613 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.797043    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4453024-52ad-45b5-9596-f6a9145f598b" containerName="cleanup"
Oct 30 22:18:24.824639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.824400    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:24.846377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.846348    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:18:24.873730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.873694    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3aa85e\""
Oct 30 22:18:24.921687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.921656    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:24.921922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.921860    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfg45\" (UniqueName: \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:24.947176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:24.947144    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3aa85e-dockercfg-zgtkw\""
Oct 30 22:18:25.022997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.022964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:25.023158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.023107    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfg45\" (UniqueName: \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:25.023721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.023702    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:25.079380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.079296    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfg45\" (UniqueName: \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\") pod \"payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:25.138577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.138547    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:25.155185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.155155    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.309326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.309291    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:25.520988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.520953    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:18:25.527526 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:18:25.527492    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3526fa1d_5bee_4a4b_9f57_059723e859c2.slice/crio-882a23c4b5573ac126262b56dd4eb0f22bb797e5311b95a350d206f114bd1801 WatchSource:0}: Error finding container 882a23c4b5573ac126262b56dd4eb0f22bb797e5311b95a350d206f114bd1801: Status 404 returned error can't find the container with id 882a23c4b5573ac126262b56dd4eb0f22bb797e5311b95a350d206f114bd1801
Oct 30 22:18:25.555905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:25.555878    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:26.270205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.270172    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:26.293952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.293918    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" event={"ID":"3526fa1d-5bee-4a4b-9f57-059723e859c2","Type":"ContainerStarted","Data":"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"}
Oct 30 22:18:26.294160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.293985    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" event={"ID":"3526fa1d-5bee-4a4b-9f57-059723e859c2","Type":"ContainerStarted","Data":"882a23c4b5573ac126262b56dd4eb0f22bb797e5311b95a350d206f114bd1801"}
Oct 30 22:18:26.294160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.294015    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:26.402398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.402336    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" podStartSLOduration=2.402318129 podStartE2EDuration="2.402318129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:26.402071552 +0000 UTC m=+2115.385978427" watchObservedRunningTime="2026-10-30 22:18:26.402318129 +0000 UTC m=+2115.386225006"
Oct 30 22:18:26.841659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:26.841619    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:27.019828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.015712    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:27.484062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.484023    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:18:27.495569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.494721    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:18:27.495569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.494850    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.543836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.543776    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-778d39d3\""
Oct 30 22:18:27.554280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.554260    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-778d39d3-dockercfg-v9svl\""
Oct 30 22:18:27.650911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.650875    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7f5s\" (UniqueName: \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.651129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.651108    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.752372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.752300    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7f5s\" (UniqueName: \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.752517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.752425    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.753088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.753070    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.806358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.806333    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7f5s\" (UniqueName: \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\") pod \"payload-processing-e2e-mt-778d39d3-5cc665b758-775sh\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") " pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:27.827182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:27.827157    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:28.200306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.200231    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:18:28.207250 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:18:28.207216    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9162831_1468_4689_a5df_3942674ebfdc.slice/crio-ae9eda6254885afeab1288735c88c3c242ab724385c089557f8b0445b96d8301 WatchSource:0}: Error finding container ae9eda6254885afeab1288735c88c3c242ab724385c089557f8b0445b96d8301: Status 404 returned error can't find the container with id ae9eda6254885afeab1288735c88c3c242ab724385c089557f8b0445b96d8301
Oct 30 22:18:28.313081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.313050    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" event={"ID":"e9162831-1468-4689-a5df-3942674ebfdc","Type":"ContainerStarted","Data":"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"}
Oct 30 22:18:28.313234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.313100    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" event={"ID":"e9162831-1468-4689-a5df-3942674ebfdc","Type":"ContainerStarted","Data":"ae9eda6254885afeab1288735c88c3c242ab724385c089557f8b0445b96d8301"}
Oct 30 22:18:28.313234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.313158    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:28.420900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.420848    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" podStartSLOduration=1.420832927 podStartE2EDuration="1.420832927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:28.420820844 +0000 UTC m=+2117.404727720" watchObservedRunningTime="2026-10-30 22:18:28.420832927 +0000 UTC m=+2117.404739793"
Oct 30 22:18:28.899090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:28.899050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:30.288372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.288338    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"]
Oct 30 22:18:30.292905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.292864    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.345981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.345948    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"]
Oct 30 22:18:30.387407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.387371    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:30.387588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.387407    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:18:30.388976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.388948    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49mv\" (UniqueName: \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.389121 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.389083    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.389266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.389248    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.490415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.490377    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.490607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.490521    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.490746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.490730    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49mv\" (UniqueName: \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.491158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.491131    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.492992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.492970    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.544075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.544012    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49mv\" (UniqueName: \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\") pod \"maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.610433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.610401    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:30.874637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.874553    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:30.996699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:30.996661    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"]
Oct 30 22:18:31.003993 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:18:31.003962    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade48ccd_0013_4cc7_8cb2_5a113dff32b3.slice/crio-59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db WatchSource:0}: Error finding container 59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db: Status 404 returned error can't find the container with id 59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db
Oct 30 22:18:31.065294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:31.065266    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:31.313975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:31.313908    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:31.340449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:31.340403    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp" event={"ID":"ade48ccd-0013-4cc7-8cb2-5a113dff32b3","Type":"ContainerStarted","Data":"59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db"}
Oct 30 22:18:31.823902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:31.823870    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:32.348886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:32.348853    5510 generic.go:356] "Generic (PLEG): container finished" podID="ade48ccd-0013-4cc7-8cb2-5a113dff32b3" containerID="2b97369ec5d29fc17612cb8773b196ff9b9c433d896728df37f366ad0a5ccd1f" exitCode=0
Oct 30 22:18:32.349280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:32.348937    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp" event={"ID":"ade48ccd-0013-4cc7-8cb2-5a113dff32b3","Type":"ContainerDied","Data":"2b97369ec5d29fc17612cb8773b196ff9b9c433d896728df37f366ad0a5ccd1f"}
Oct 30 22:18:32.459660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:32.459614    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.062265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.062226    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.227814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.227755    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.451117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.451089    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:18:33.451700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.451655    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" containerID="cri-o://ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5" gracePeriod=30
Oct 30 22:18:33.468314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.468280    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:33.644854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.641803    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:33.792346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.792305    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\") pod \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") "
Oct 30 22:18:33.792528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.792501    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\") pod \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") "
Oct 30 22:18:33.792740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.792708    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca" pod "ade48ccd-0013-4cc7-8cb2-5a113dff32b3" (UID: "ade48ccd-0013-4cc7-8cb2-5a113dff32b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:33.792872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.792721    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\" (UniqueName: \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\") pod \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\" (UID: \"ade48ccd-0013-4cc7-8cb2-5a113dff32b3\") "
Oct 30 22:18:33.793494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.793466    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:33.795300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.795274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca" pod "ade48ccd-0013-4cc7-8cb2-5a113dff32b3" (UID: "ade48ccd-0013-4cc7-8cb2-5a113dff32b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:33.795419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.795313    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv" pod "ade48ccd-0013-4cc7-8cb2-5a113dff32b3" (UID: "ade48ccd-0013-4cc7-8cb2-5a113dff32b3"). InnerVolumeSpecName "kube-api-access-t49mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:33.895234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.895154    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t49mv\" (UniqueName: \"kubernetes.io/projected/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-kube-api-access-t49mv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:33.895234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:33.895183    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ade48ccd-0013-4cc7-8cb2-5a113dff32b3-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:34.374146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:34.374108    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp" event={"ID":"ade48ccd-0013-4cc7-8cb2-5a113dff32b3","Type":"ContainerDied","Data":"59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db"}
Oct 30 22:18:34.374146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:34.374135    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"
Oct 30 22:18:34.374420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:34.374170    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59467d29a10721c3824f88c8d4ca046f9cbbeb5b3bea04928479c082ef0365db"
Oct 30 22:18:35.281628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.281595    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:35.746803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.741456    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"]
Oct 30 22:18:35.746803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.742985    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ade48ccd-0013-4cc7-8cb2-5a113dff32b3" containerName="revoke-keys"
Oct 30 22:18:35.746803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.743007    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade48ccd-0013-4cc7-8cb2-5a113dff32b3" containerName="revoke-keys"
Oct 30 22:18:35.746803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.743360    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade48ccd-0013-4cc7-8cb2-5a113dff32b3" containerName="revoke-keys"
Oct 30 22:18:35.748838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.748811    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:35.795314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.795273    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"]
Oct 30 22:18:35.837396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.837363    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:35.837561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.837450    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:18:35.918187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.918126    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:35.922599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.922575    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8vs\" (UniqueName: \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:35.922800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.922769    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:35.922930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:35.922911    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.023995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.023910    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.024140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.024030    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8vs\" (UniqueName: \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.024241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.024187    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.024683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.024661    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.026564 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.026544    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.078227 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.078201    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8vs\" (UniqueName: \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.363102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.363007    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:36.743448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:36.743413    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"]
Oct 30 22:18:36.750296 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:18:36.750263    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd7094d_d0c7_4d6f_9d40_89ffb9e8ac0f.slice/crio-65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2 WatchSource:0}: Error finding container 65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2: Status 404 returned error can't find the container with id 65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2
Oct 30 22:18:37.305906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.305878    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:18:37.401651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.401566    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t" event={"ID":"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f","Type":"ContainerStarted","Data":"d8863212402a375e3f53938c6c6fa4fae676d95c3691ed7658b573583a38e43e"}
Oct 30 22:18:37.401651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.401615    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t" event={"ID":"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f","Type":"ContainerStarted","Data":"65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2"}
Oct 30 22:18:37.538245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.535865    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t" podStartSLOduration=2.5358463159999998 podStartE2EDuration="2.535846316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:37.534047688 +0000 UTC m=+2126.517954564" watchObservedRunningTime="2026-10-30 22:18:37.535846316 +0000 UTC m=+2126.519753194"
Oct 30 22:18:37.618012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.617966    5510 patch_prober.go:29] interesting pod/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:37.618155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:37.618032    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:38.411038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.410999    5510 generic.go:356] "Generic (PLEG): container finished" podID="6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" containerID="d8863212402a375e3f53938c6c6fa4fae676d95c3691ed7658b573583a38e43e" exitCode=0
Oct 30 22:18:38.411452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.411080    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t" event={"ID":"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f","Type":"ContainerDied","Data":"d8863212402a375e3f53938c6c6fa4fae676d95c3691ed7658b573583a38e43e"}
Oct 30 22:18:38.747215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.747185    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:18:38.807671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.807637    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:38.867485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867451    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867509    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867539    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867605    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867815    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:38.867930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867822    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.867930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867913    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:38.868096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867945    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:38.868096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867970    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.868096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868052    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\") pod \"8fd46d7c-c737-41e2-a61d-f36e297bb032\" (UID: \"8fd46d7c-c737-41e2-a61d-f36e297bb032\") "
Oct 30 22:18:38.868270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.867970    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:38.868592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868547    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:38.868892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868867    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.868991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868898    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd46d7c-c737-41e2-a61d-f36e297bb032-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.868991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868913    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.868991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868931    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.868991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.868946    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.870142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.870112    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:38.870355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.870329    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:38.870475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.870394    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "kube-api-access-kffwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:38.870475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.870426    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo" pod "8fd46d7c-c737-41e2-a61d-f36e297bb032" (UID: "8fd46d7c-c737-41e2-a61d-f36e297bb032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:38.970721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.970622    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.970721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.970655    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.970721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.970668    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:38.970721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:38.970683    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kffwt\" (UniqueName: \"kubernetes.io/projected/8fd46d7c-c737-41e2-a61d-f36e297bb032-kube-api-access-kffwt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:39.032109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.032077    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.168206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.168171    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:18:39.168436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.168410    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy" containerID="cri-o://aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324" gracePeriod=30
Oct 30 22:18:39.237878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.237795    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.362806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.361999    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:18:39.429288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.427875    5510 generic.go:356] "Generic (PLEG): container finished" podID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerID="ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5" exitCode=0
Oct 30 22:18:39.429288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.428064    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerDied","Data":"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"}
Oct 30 22:18:39.429288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.428139    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr" event={"ID":"8fd46d7c-c737-41e2-a61d-f36e297bb032","Type":"ContainerDied","Data":"692660f0c5e61f5153ecbf01f27e892b25bcc39ba2eeab9fd805c54ac8fcbf9a"}
Oct 30 22:18:39.429288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.428168    5510 scope.go:122] "RemoveContainer" containerID="ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"
Oct 30 22:18:39.429288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.428182    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"
Oct 30 22:18:39.450148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.450079    5510 scope.go:122] "RemoveContainer" containerID="6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"
Oct 30 22:18:39.463545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.463520    5510 scope.go:122] "RemoveContainer" containerID="ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"
Oct 30 22:18:39.463946 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:39.463924    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5\": container with ID starting with ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5 not found: ID does not exist" containerID="ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"
Oct 30 22:18:39.464067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.464048    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5"} err="failed to get container status \"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5\": rpc error: code = NotFound desc = could not find container \"ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5\": container with ID starting with ed863710792a006c3cfd6d7d6cbbcc1be1f694a04a4a1c1cf30b050ffc453ab5 not found: ID does not exist"
Oct 30 22:18:39.464134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.464127    5510 scope.go:122] "RemoveContainer" containerID="6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"
Oct 30 22:18:39.464480 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:39.464449    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4\": container with ID starting with 6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4 not found: ID does not exist" containerID="6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"
Oct 30 22:18:39.464558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.464490    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4"} err="failed to get container status \"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4\": rpc error: code = NotFound desc = could not find container \"6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4\": container with ID starting with 6839967e5df5cd3815514a6632c5e131c4ea3cd097277532c9cf239dbc0532b4 not found: ID does not exist"
Oct 30 22:18:39.585753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.585721    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:18:39.603842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.603805    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-263513-openshift-default-7ccd864455-n5cqr"]
Oct 30 22:18:39.649446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.649417    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:39.703277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.703237    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\") pod \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") "
Oct 30 22:18:39.703681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.703629    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca" pod "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" (UID: "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:39.703992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.703907    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\") pod \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") "
Oct 30 22:18:39.704110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.704080    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\" (UniqueName: \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\") pod \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\" (UID: \"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f\") "
Oct 30 22:18:39.706341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.705875    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:39.710770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.708036    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca" pod "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" (UID: "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:39.711271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.711249    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs" pod "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" (UID: "6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f"). InnerVolumeSpecName "kube-api-access-zh8vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:39.807074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.806999    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:39.807074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.807031    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8vs\" (UniqueName: \"kubernetes.io/projected/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f-kube-api-access-zh8vs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:39.854401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:39.854370    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:40.249120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.249089    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:18:40.263754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.263715    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 22:18:40.278928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.278895    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 22:18:40.441188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.441162    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"
Oct 30 22:18:40.441188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.441178    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t" event={"ID":"6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f","Type":"ContainerDied","Data":"65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2"}
Oct 30 22:18:40.441937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.441227    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65334a2038ac39cdec8c3ebb2215b3e4bc809dd9280a86cfc3b3a9bcdefa53d2"
Oct 30 22:18:40.457338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:40.457308    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:41.060609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:41.060572    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:41.129913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:41.129875    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" path="/var/lib/kubelet/pods/8fd46d7c-c737-41e2-a61d-f36e297bb032/volumes"
Oct 30 22:18:41.241368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:41.241339    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:42.164421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:42.164390    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:18:42.164884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:42.164577    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy" containerID="cri-o://15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691" gracePeriod=30
Oct 30 22:18:42.206420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:42.206383    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:42.394433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:42.394396    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:43.327949 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:43.326364    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:18:44.184494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.184466    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.337372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.337334    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:44.462559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.462529    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:18:44.526807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.509242    5510 generic.go:356] "Generic (PLEG): container finished" podID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerID="aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324" exitCode=0
Oct 30 22:18:44.526807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.509457    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerDied","Data":"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"}
Oct 30 22:18:44.526807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.509515    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n" event={"ID":"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7","Type":"ContainerDied","Data":"02f285d9a878cce92af0c1136eb1c7daf8fe63fac2b2c10e98d37df74b10d9d9"}
Oct 30 22:18:44.526807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.509545    5510 scope.go:122] "RemoveContainer" containerID="aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"
Oct 30 22:18:44.526807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.509739    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"
Oct 30 22:18:44.591988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.591952    5510 scope.go:122] "RemoveContainer" containerID="c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"
Oct 30 22:18:44.609499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.609474    5510 scope.go:122] "RemoveContainer" containerID="aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"
Oct 30 22:18:44.610067 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:44.610043    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324\": container with ID starting with aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324 not found: ID does not exist" containerID="aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"
Oct 30 22:18:44.610232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.610203    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324"} err="failed to get container status \"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324\": rpc error: code = NotFound desc = could not find container \"aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324\": container with ID starting with aea95552d5da2914651e73b4acb6ae3c6ba29478d8a78c123cb6adf5aba6f324 not found: ID does not exist"
Oct 30 22:18:44.610387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.610234    5510 scope.go:122] "RemoveContainer" containerID="c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"
Oct 30 22:18:44.610545 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:44.610524    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38\": container with ID starting with c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38 not found: ID does not exist" containerID="c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"
Oct 30 22:18:44.610626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.610554    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38"} err="failed to get container status \"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38\": rpc error: code = NotFound desc = could not find container \"c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38\": container with ID starting with c5c53710fc574894d7c15d7e5bedf74c599b3ac70872417880c89cbd3926fc38 not found: ID does not exist"
Oct 30 22:18:44.626257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.626224    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.626488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.626467    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.626679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.626661    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.626903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.626883    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.627058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.627043    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.627262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.627243    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.627464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.627444    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.627659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.627641    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.627853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.627834    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\") pod \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\" (UID: \"2be2cac5-c8d2-4d14-b775-ef3f9ed372f7\") "
Oct 30 22:18:44.630608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.629021    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:44.630608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.629322    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:44.630608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.629477    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:44.630608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.629609    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:44.630608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.629734    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:44.634514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.634423    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:44.634514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.634459    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:44.634514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.634475    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:44.634959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.634935    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6" pod "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" (UID: "2be2cac5-c8d2-4d14-b775-ef3f9ed372f7"). InnerVolumeSpecName "kube-api-access-5j4d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:44.729894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729866    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.729894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729893    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729903    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729912    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729921    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729930    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4d6\" (UniqueName: \"kubernetes.io/projected/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-kube-api-access-5j4d6\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729940    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729948    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.730103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.729957    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:44.912414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.912380    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:18:44.928288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.928256    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:18:44.937684 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:44.937658    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-411ff1-openshift-default-65c447db45-twm5n"]
Oct 30 22:18:45.127281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:45.127214    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" path="/var/lib/kubelet/pods/2be2cac5-c8d2-4d14-b775-ef3f9ed372f7/volumes"
Oct 30 22:18:46.018885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:46.018851    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.157304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.157260    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 22:18:47.458584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.458548    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:18:47.535142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.535108    5510 generic.go:356] "Generic (PLEG): container finished" podID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerID="15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691" exitCode=0
Oct 30 22:18:47.535326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.535173    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" event={"ID":"b51537bc-dd14-43a0-964e-dae730ad5fb4","Type":"ContainerDied","Data":"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"}
Oct 30 22:18:47.535326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.535241    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt" event={"ID":"b51537bc-dd14-43a0-964e-dae730ad5fb4","Type":"ContainerDied","Data":"a4109a4b0710d97c49252d9b78fac7d08faf38f3baca4102b5a0423966ac6183"}
Oct 30 22:18:47.535449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.535274    5510 scope.go:122] "RemoveContainer" containerID="15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"
Oct 30 22:18:47.535512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.535454    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"
Oct 30 22:18:47.554061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.553993    5510 scope.go:122] "RemoveContainer" containerID="15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"
Oct 30 22:18:47.554382 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:18:47.554331    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691\": container with ID starting with 15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691 not found: ID does not exist" containerID="15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"
Oct 30 22:18:47.554494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.554373    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691"} err="failed to get container status \"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691\": rpc error: code = NotFound desc = could not find container \"15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691\": container with ID starting with 15f00e9e250e523fefd636a93db84744f81e07ddc8878f2c877a5d6276501691 not found: ID does not exist"
Oct 30 22:18:47.594119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594087    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594215    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594315    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594355    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594363    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.594423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594394    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594427    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594462    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594506    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594578    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.594709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594595    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.594709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594620    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\") pod \"b51537bc-dd14-43a0-964e-dae730ad5fb4\" (UID: \"b51537bc-dd14-43a0-964e-dae730ad5fb4\") "
Oct 30 22:18:47.594898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594838    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.594965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.594928    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:47.595361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.595336    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.595464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.595367    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.595464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.595388    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51537bc-dd14-43a0-964e-dae730ad5fb4-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.595464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.595403    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.595464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.595418    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.596717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.596679    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "kube-api-access-284cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:47.596827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.596712    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:47.596919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.596901    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.596987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.596950    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token" pod "b51537bc-dd14-43a0-964e-dae730ad5fb4" (UID: "b51537bc-dd14-43a0-964e-dae730ad5fb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:47.623742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.623710    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.697307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.697236    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.697307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.697264    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.697307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.697276    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284cw\" (UniqueName: \"kubernetes.io/projected/b51537bc-dd14-43a0-964e-dae730ad5fb4-kube-api-access-284cw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.697307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.697286    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51537bc-dd14-43a0-964e-dae730ad5fb4-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.769387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.769349    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.954500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.954410    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:18:47.957675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:47.957648    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-59nvt"]
Oct 30 22:18:48.241245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241217    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"]
Oct 30 22:18:48.241920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241905    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.241981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241922    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.241981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241934    5510 cpu_manager.go:414] "containerMap: removing container" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.241981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241939    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.241981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241954    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:18:48.241981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241959    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:18:48.242139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.241999    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy"
Oct 30 22:18:48.242139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242005    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy"
Oct 30 22:18:48.242139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242044    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" containerName="revoke-keys"
Oct 30 22:18:48.242139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242050    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" containerName="revoke-keys"
Oct 30 22:18:48.242255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242167    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" containerName="revoke-keys"
Oct 30 22:18:48.242255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242177    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:18:48.242255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242184    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.242255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242193    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd46d7c-c737-41e2-a61d-f36e297bb032" containerName="istio-proxy"
Oct 30 22:18:48.242255 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.242202    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" containerName="istio-proxy"
Oct 30 22:18:48.247362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.247343    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.299377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.299347    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"]
Oct 30 22:18:48.299653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.299610    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:18:48.299763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.299682    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:48.412451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.412416    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.412662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.412644    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dzj\" (UniqueName: \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.412736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.412712    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.514818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.514705    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dzj\" (UniqueName: \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.514818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.514762    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.515053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.514861    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.515439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.515418    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.517145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.517129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.568846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.568814    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dzj\" (UniqueName: \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\") pod \"maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:48.879851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:48.879050    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:49.128066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:49.128024    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51537bc-dd14-43a0-964e-dae730ad5fb4" path="/var/lib/kubelet/pods/b51537bc-dd14-43a0-964e-dae730ad5fb4/volumes"
Oct 30 22:18:49.311136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:49.311097    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"]
Oct 30 22:18:49.315270 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:18:49.315239    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cfb3ee_0703_4440_9bb0_c488f30d2162.slice/crio-7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005 WatchSource:0}: Error finding container 7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005: Status 404 returned error can't find the container with id 7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005
Oct 30 22:18:49.553077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:49.553040    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c" event={"ID":"d5cfb3ee-0703-4440-9bb0-c488f30d2162","Type":"ContainerStarted","Data":"7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005"}
Oct 30 22:18:50.216622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:50.216588    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:50.370287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:50.370248    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:50.522998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:50.522910    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:50.570357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:50.570323    5510 generic.go:356] "Generic (PLEG): container finished" podID="d5cfb3ee-0703-4440-9bb0-c488f30d2162" containerID="8873fed740a48b57ac668387534f78bad4388c7aa83a1ccbff0655dd107b063f" exitCode=0
Oct 30 22:18:50.570549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:50.570379    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c" event={"ID":"d5cfb3ee-0703-4440-9bb0-c488f30d2162","Type":"ContainerDied","Data":"8873fed740a48b57ac668387534f78bad4388c7aa83a1ccbff0655dd107b063f"}
Oct 30 22:18:51.297090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.297052    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:51.728803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.728752    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:51.758224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.758196    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\" (UniqueName: \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\") pod \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") "
Oct 30 22:18:51.758380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.758259    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\") pod \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") "
Oct 30 22:18:51.758457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.758387    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\") pod \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\" (UID: \"d5cfb3ee-0703-4440-9bb0-c488f30d2162\") "
Oct 30 22:18:51.758797 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.758751    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca" pod "d5cfb3ee-0703-4440-9bb0-c488f30d2162" (UID: "d5cfb3ee-0703-4440-9bb0-c488f30d2162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:51.758953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.758931    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:51.760302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.760273    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca" pod "d5cfb3ee-0703-4440-9bb0-c488f30d2162" (UID: "d5cfb3ee-0703-4440-9bb0-c488f30d2162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:51.760648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.760622    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj" pod "d5cfb3ee-0703-4440-9bb0-c488f30d2162" (UID: "d5cfb3ee-0703-4440-9bb0-c488f30d2162"). InnerVolumeSpecName "kube-api-access-z9dzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:51.860333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.860252    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9dzj\" (UniqueName: \"kubernetes.io/projected/d5cfb3ee-0703-4440-9bb0-c488f30d2162-kube-api-access-z9dzj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:51.860333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:51.860294    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cfb3ee-0703-4440-9bb0-c488f30d2162-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:52.158124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:52.158026    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing" containerID="cri-o://822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44" gracePeriod=25
Oct 30 22:18:52.586258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:52.586212    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c" event={"ID":"d5cfb3ee-0703-4440-9bb0-c488f30d2162","Type":"ContainerDied","Data":"7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005"}
Oct 30 22:18:52.586462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:52.586280    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7054b5820fc67d0e65d70cd2127bc98559d89225677c8337e6e0e1d1d5aae005"
Oct 30 22:18:52.586462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:52.586235    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"
Oct 30 22:18:55.273325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.273274    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" containerID="cri-o://a884b1b826da7d7b6bac49cfdbddd4b240869ac06a4f7cea4f4fbb462b77a450" gracePeriod=45
Oct 30 22:18:55.286542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.286503    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" containerID="cri-o://8826c335763254851d43b9b50dfd77ec65a08451daae52ff224a63eef4ee8a20" gracePeriod=45
Oct 30 22:18:55.292591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.292557    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused"
Oct 30 22:18:55.301294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.301260    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" containerID="cri-o://6014c6f85254931b0ae557a4545146610cc5773028369fe8704f0307200ad6fa" gracePeriod=45
Oct 30 22:18:55.305254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.305225    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": EOF"
Oct 30 22:18:55.492715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.490708    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused"
Oct 30 22:18:55.614200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.614172    5510 generic.go:356] "Generic (PLEG): container finished" podID="99b7df0a-69af-4986-8824-3376a157b895" containerID="8826c335763254851d43b9b50dfd77ec65a08451daae52ff224a63eef4ee8a20" exitCode=0
Oct 30 22:18:55.614322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.614246    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" event={"ID":"99b7df0a-69af-4986-8824-3376a157b895","Type":"ContainerDied","Data":"8826c335763254851d43b9b50dfd77ec65a08451daae52ff224a63eef4ee8a20"}
Oct 30 22:18:55.614322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.614288    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5" event={"ID":"99b7df0a-69af-4986-8824-3376a157b895","Type":"ContainerDied","Data":"ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195"}
Oct 30 22:18:55.614322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.614299    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad937297b14b59546939447809aa129116e42dfed78bf54c38898d4836fa4195"
Oct 30 22:18:55.616055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.616028    5510 generic.go:356] "Generic (PLEG): container finished" podID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerID="6014c6f85254931b0ae557a4545146610cc5773028369fe8704f0307200ad6fa" exitCode=0
Oct 30 22:18:55.616172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.616069    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" event={"ID":"6a278fdd-ff55-4c0f-9c6e-e64469fc241f","Type":"ContainerDied","Data":"6014c6f85254931b0ae557a4545146610cc5773028369fe8704f0307200ad6fa"}
Oct 30 22:18:55.616172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.616108    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6" event={"ID":"6a278fdd-ff55-4c0f-9c6e-e64469fc241f","Type":"ContainerDied","Data":"b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6"}
Oct 30 22:18:55.616172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.616128    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e8f63a82b1ed7a0bb338fd578cdfdffa41bb2d70d5b2b3c51f64e673dcb0d6"
Oct 30 22:18:55.617569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.617550    5510 generic.go:356] "Generic (PLEG): container finished" podID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerID="a884b1b826da7d7b6bac49cfdbddd4b240869ac06a4f7cea4f4fbb462b77a450" exitCode=0
Oct 30 22:18:55.617656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.617622    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" event={"ID":"d1262f04-8f4e-4c25-87ca-99dc7013c6db","Type":"ContainerDied","Data":"a884b1b826da7d7b6bac49cfdbddd4b240869ac06a4f7cea4f4fbb462b77a450"}
Oct 30 22:18:55.634031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.634009    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:18:55.638123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.638099    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:18:55.642092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.642075    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:18:55.721963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.721931    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.721975    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.721995    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722030    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\" (UniqueName: \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722057    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722076    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.722145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722101    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722164    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722201    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722247    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722255    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722387    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722397    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722401    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.722539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722506    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722627    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\" (UniqueName: \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722641    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722673    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\" (UniqueName: \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722802    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722845    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\") pod \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\" (UID: \"d1262f04-8f4e-4c25-87ca-99dc7013c6db\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722901    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\") pod \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\" (UID: \"6a278fdd-ff55-4c0f-9c6e-e64469fc241f\") "
Oct 30 22:18:55.723115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722975    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\") pod \"99b7df0a-69af-4986-8824-3376a157b895\" (UID: \"99b7df0a-69af-4986-8824-3376a157b895\") "
Oct 30 22:18:55.723635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.723609    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.723916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.723890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.724053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.723913    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.724318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724297    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724348    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724364    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724378    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724393    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724406    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724419    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724434    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.724765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.722553    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.724997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:55.724997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.724944    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "kube-api-access-vjvxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:55.725131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.725009    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.725171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.725122    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.725171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.725142    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt" pod "99b7df0a-69af-4986-8824-3376a157b895" (UID: "99b7df0a-69af-4986-8824-3376a157b895"). InnerVolumeSpecName "kube-api-access-7b8gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:55.725653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.725628    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:55.726238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.726217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:55.726410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.726390    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8" pod "d1262f04-8f4e-4c25-87ca-99dc7013c6db" (UID: "d1262f04-8f4e-4c25-87ca-99dc7013c6db"). InnerVolumeSpecName "kube-api-access-7p5q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:55.726849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.726829    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs" pod "6a278fdd-ff55-4c0f-9c6e-e64469fc241f" (UID: "6a278fdd-ff55-4c0f-9c6e-e64469fc241f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:55.825590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825553    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1262f04-8f4e-4c25-87ca-99dc7013c6db-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825587    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825601    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8gt\" (UniqueName: \"kubernetes.io/projected/99b7df0a-69af-4986-8824-3376a157b895-kube-api-access-7b8gt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825616    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1262f04-8f4e-4c25-87ca-99dc7013c6db-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825629    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5q8\" (UniqueName: \"kubernetes.io/projected/d1262f04-8f4e-4c25-87ca-99dc7013c6db-kube-api-access-7p5q8\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825642    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825654    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825669    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvxq\" (UniqueName: \"kubernetes.io/projected/6a278fdd-ff55-4c0f-9c6e-e64469fc241f-kube-api-access-vjvxq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825681    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b7df0a-69af-4986-8824-3376a157b895-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:55.825836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:55.825693    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b7df0a-69af-4986-8824-3376a157b895-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:56.012510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.012481    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:56.625539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.625449    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"
Oct 30 22:18:56.625539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.625482    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"
Oct 30 22:18:56.625539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.625479    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth" event={"ID":"d1262f04-8f4e-4c25-87ca-99dc7013c6db","Type":"ContainerDied","Data":"6808d7942a9b9f02d1ca01ef379270593a60a39b5227bbc036d00ccf3c027ba3"}
Oct 30 22:18:56.625539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.625454    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"
Oct 30 22:18:56.625539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.625538    5510 scope.go:122] "RemoveContainer" containerID="a884b1b826da7d7b6bac49cfdbddd4b240869ac06a4f7cea4f4fbb462b77a450"
Oct 30 22:18:56.826629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.826593    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:18:56.832376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.832348    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/distinct-w3-f776cb-kserve-7dd6d4c8df-8jsth"]
Oct 30 22:18:56.936194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.936101    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 22:18:56.939595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:56.939566    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/premium-w3-f776cb-kserve-67dcc47b4b-8lzt5"]
Oct 30 22:18:57.038574 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:57.038522    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 22:18:57.043364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:57.041826    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f776cb/simulator-w3-f776cb-kserve-c5d98f599-fmtk6"]
Oct 30 22:18:57.128957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:57.128917    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" path="/var/lib/kubelet/pods/6a278fdd-ff55-4c0f-9c6e-e64469fc241f/volumes"
Oct 30 22:18:57.129402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:57.129385    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b7df0a-69af-4986-8824-3376a157b895" path="/var/lib/kubelet/pods/99b7df0a-69af-4986-8824-3376a157b895/volumes"
Oct 30 22:18:57.129816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:57.129775    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" path="/var/lib/kubelet/pods/d1262f04-8f4e-4c25-87ca-99dc7013c6db/volumes"
Oct 30 22:18:59.677242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:59.677208    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:18:59.779223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:18:59.779184    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:19:00.027183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:00.027145    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:00.231146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:00.231108    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:00.379548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:00.379462    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:00.725918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:00.725875    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:19:01.727868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:01.727835    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:01.878101 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:01.878064    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:02.048579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:02.048499    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:02.197450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:02.197369    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:02.802976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:02.802940    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:04.677847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:04.677809    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" podUID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" containerName="payload-processing" containerID="cri-o://fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb" gracePeriod=25
Oct 30 22:19:04.780235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:04.780166    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" podUID="1297551d-b906-442e-91a7-2d41dff03574" containerName="payload-pre-processing" containerID="cri-o://f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592" gracePeriod=25
Oct 30 22:19:04.944890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:04.944824    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:19:05.018156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.018121    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\" (UniqueName: \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\") pod \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") "
Oct 30 22:19:05.018365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.018182    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\") pod \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\" (UID: \"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c\") "
Oct 30 22:19:05.018955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.018921    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume" pod "9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" (UID: "9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:05.020636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.020597    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt" pod "9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" (UID: "9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c"). InnerVolumeSpecName "kube-api-access-5vmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:05.062543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.062506    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:19:05.119260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.119225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\" (UniqueName: \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\") pod \"1297551d-b906-442e-91a7-2d41dff03574\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") "
Oct 30 22:19:05.119506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.119486    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\") pod \"1297551d-b906-442e-91a7-2d41dff03574\" (UID: \"1297551d-b906-442e-91a7-2d41dff03574\") "
Oct 30 22:19:05.120096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.120071    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmvt\" (UniqueName: \"kubernetes.io/projected/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-kube-api-access-5vmvt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.120230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.120098    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.120230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.120160    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume" pod "1297551d-b906-442e-91a7-2d41dff03574" (UID: "1297551d-b906-442e-91a7-2d41dff03574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:05.122145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.122118    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x" pod "1297551d-b906-442e-91a7-2d41dff03574" (UID: "1297551d-b906-442e-91a7-2d41dff03574"). InnerVolumeSpecName "kube-api-access-f5w8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:05.221646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.221614    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1297551d-b906-442e-91a7-2d41dff03574-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.221646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.221643    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5w8x\" (UniqueName: \"kubernetes.io/projected/1297551d-b906-442e-91a7-2d41dff03574-kube-api-access-f5w8x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.689353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.689259    5510 generic.go:356] "Generic (PLEG): container finished" podID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" containerID="fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb" exitCode=0
Oct 30 22:19:05.689353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.689304    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"
Oct 30 22:19:05.689353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.689343    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" event={"ID":"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c","Type":"ContainerDied","Data":"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"}
Oct 30 22:19:05.689939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.689398    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9" event={"ID":"9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c","Type":"ContainerDied","Data":"5da483bd9d55e2ef87332b7c5951a668cec1761829934e801333b2301abd3d0a"}
Oct 30 22:19:05.689939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.689432    5510 scope.go:122] "RemoveContainer" containerID="fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"
Oct 30 22:19:05.690910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.690879    5510 generic.go:356] "Generic (PLEG): container finished" podID="1297551d-b906-442e-91a7-2d41dff03574" containerID="f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592" exitCode=0
Oct 30 22:19:05.691017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.690927    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"
Oct 30 22:19:05.691017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.690954    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" event={"ID":"1297551d-b906-442e-91a7-2d41dff03574","Type":"ContainerDied","Data":"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"}
Oct 30 22:19:05.691017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.691004    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg" event={"ID":"1297551d-b906-442e-91a7-2d41dff03574","Type":"ContainerDied","Data":"be6d5e00312f031af9c3108893a5be74221f8da2236253552bc3e2ff897fbf9e"}
Oct 30 22:19:05.706032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.706013    5510 scope.go:122] "RemoveContainer" containerID="fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"
Oct 30 22:19:05.706348 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:05.706328    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb\": container with ID starting with fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb not found: ID does not exist" containerID="fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"
Oct 30 22:19:05.706397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.706356    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb"} err="failed to get container status \"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb\": rpc error: code = NotFound desc = could not find container \"fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb\": container with ID starting with fc65df68380d4402ea1d4ddd272978c672e56c892140bba246fb5355720cc1fb not found: ID does not exist"
Oct 30 22:19:05.706397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.706376    5510 scope.go:122] "RemoveContainer" containerID="f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"
Oct 30 22:19:05.715761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.715738    5510 scope.go:122] "RemoveContainer" containerID="f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"
Oct 30 22:19:05.716106 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:05.716076    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592\": container with ID starting with f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592 not found: ID does not exist" containerID="f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"
Oct 30 22:19:05.716196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.716115    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592"} err="failed to get container status \"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592\": rpc error: code = NotFound desc = could not find container \"f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592\": container with ID starting with f08ae6cf0de3cb109e132383e774a9a4f73e9e2ebab5a0183a97636c089fb592 not found: ID does not exist"
Oct 30 22:19:05.842517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.842476    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:19:05.888257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.888225    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5fef81-67775f8c6d-m2hj9"]
Oct 30 22:19:05.988979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.988940    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:19:05.992820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:05.992776    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5fef81-58548778cf-4srxg"]
Oct 30 22:19:06.242033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:06.241918    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.389652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:06.389603    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.810974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:06.810938    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:07.126959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:07.126881    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1297551d-b906-442e-91a7-2d41dff03574" path="/var/lib/kubelet/pods/1297551d-b906-442e-91a7-2d41dff03574/volumes"
Oct 30 22:19:07.127412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:07.127388    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" path="/var/lib/kubelet/pods/9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c/volumes"
Oct 30 22:19:09.139337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:09.139285    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:09.288468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:09.288428    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:09.472302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:09.472267    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:19:09.472511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:09.472489    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy" containerID="cri-o://7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511" gracePeriod=30
Oct 30 22:19:09.495139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:09.495110    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:10.367399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.367322    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:10.698581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.698502    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:19:10.900692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.900654    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:19:10.901311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901294    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901313    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901327    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1297551d-b906-442e-91a7-2d41dff03574" containerName="payload-pre-processing"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901332    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1297551d-b906-442e-91a7-2d41dff03574" containerName="payload-pre-processing"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901350    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901356    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:19:10.901373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901373    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901378    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901386    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" containerName="payload-processing"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901391    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" containerName="payload-processing"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901400    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d5cfb3ee-0703-4440-9bb0-c488f30d2162" containerName="revoke-keys"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901405    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cfb3ee-0703-4440-9bb0-c488f30d2162" containerName="revoke-keys"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901426    5510 cpu_manager.go:414] "containerMap: removing container" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901430    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901511    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a278fdd-ff55-4c0f-9c6e-e64469fc241f" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901520    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1297551d-b906-442e-91a7-2d41dff03574" containerName="payload-pre-processing"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901528    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be2cac5-c8d2-4d14-b775-ef3f9ed372f7" containerName="istio-proxy"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901539    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1262f04-8f4e-4c25-87ca-99dc7013c6db" containerName="main"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901544    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c35fd26-9a8b-4dc8-85a1-e8889ca6ec9c" containerName="payload-processing"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901551    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cfb3ee-0703-4440-9bb0-c488f30d2162" containerName="revoke-keys"
Oct 30 22:19:10.901629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.901559    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b7df0a-69af-4986-8824-3376a157b895" containerName="main"
Oct 30 22:19:10.906322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.906300    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:10.961851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:10.958514    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-edf6d5-openshift-default-dockercfg-wkj8w\""
Oct 30 22:19:11.008501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.008458    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:19:11.012735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.012694    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:11.021290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021255    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.021723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.021893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021797    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.021893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021832    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.022050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021893    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.022050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.021976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.022168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.022050    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.022168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.022110    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.022168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.022159    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6wh\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.125682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.125641    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.125989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.125967    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126156    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126278    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126413    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126540    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126674    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126833 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126818    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.126986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126961    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.127060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126973    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6wh\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.127553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.127533    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.127685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.127566    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.127792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.127708    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.127891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.126718    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.129368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.129323    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.132009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.131914    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.180890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.180851    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.181125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.181104    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6wh\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\") pod \"e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") " pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.320591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.320503    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-edf6d5-openshift-default-dockercfg-wkj8w\""
Oct 30 22:19:11.322731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.322706    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:11.323564 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.323552    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:11.664012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.663935    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:11.756480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.756443    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:19:11.762100 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:19:11.762055    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9cebe2_8a6f_41c7_b619_c84dc4e243f1.slice/crio-81c78d8b0ad646562c34fb9f9976b5cd9bfd0dbe1b7cee809954cb11b7e525bf WatchSource:0}: Error finding container 81c78d8b0ad646562c34fb9f9976b5cd9bfd0dbe1b7cee809954cb11b7e525bf: Status 404 returned error can't find the container with id 81c78d8b0ad646562c34fb9f9976b5cd9bfd0dbe1b7cee809954cb11b7e525bf
Oct 30 22:19:11.765224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.765185    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:19:11.765325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.765275    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:19:11.765325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:11.765319    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:19:12.430373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.430315    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 22:19:12.546152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.546120    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\") pod \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") "
Oct 30 22:19:12.546341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.546171    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\" (UniqueName: \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\") pod \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\" (UID: \"a3a70268-c93c-4ef6-ae1c-bae0636c20f5\") "
Oct 30 22:19:12.546552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.546521    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume" pod "a3a70268-c93c-4ef6-ae1c-bae0636c20f5" (UID: "a3a70268-c93c-4ef6-ae1c-bae0636c20f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:12.548214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.548192    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69" pod "a3a70268-c93c-4ef6-ae1c-bae0636c20f5" (UID: "a3a70268-c93c-4ef6-ae1c-bae0636c20f5"). InnerVolumeSpecName "kube-api-access-spb69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:12.648104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.648060    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.648104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.648101    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spb69\" (UniqueName: \"kubernetes.io/projected/a3a70268-c93c-4ef6-ae1c-bae0636c20f5-kube-api-access-spb69\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.746972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.746938    5510 generic.go:356] "Generic (PLEG): container finished" podID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerID="822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44" exitCode=0
Oct 30 22:19:12.747453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.746979    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"
Oct 30 22:19:12.747453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.747025    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" event={"ID":"a3a70268-c93c-4ef6-ae1c-bae0636c20f5","Type":"ContainerDied","Data":"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"}
Oct 30 22:19:12.747453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.747083    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn" event={"ID":"a3a70268-c93c-4ef6-ae1c-bae0636c20f5","Type":"ContainerDied","Data":"fcd6d086b7c4fef97657ae480e489f359f2d24b5af9a3bf12e28c816fdfe5eda"}
Oct 30 22:19:12.747453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.747113    5510 scope.go:122] "RemoveContainer" containerID="822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"
Oct 30 22:19:12.749414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.749389    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerStarted","Data":"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"}
Oct 30 22:19:12.749527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.749435    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerStarted","Data":"81c78d8b0ad646562c34fb9f9976b5cd9bfd0dbe1b7cee809954cb11b7e525bf"}
Oct 30 22:19:12.759142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.759124    5510 scope.go:122] "RemoveContainer" containerID="822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"
Oct 30 22:19:12.759403 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:12.759380    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44\": container with ID starting with 822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44 not found: ID does not exist" containerID="822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"
Oct 30 22:19:12.759476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.759412    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44"} err="failed to get container status \"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44\": rpc error: code = NotFound desc = could not find container \"822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44\": container with ID starting with 822ec61ca227bf94cdd5451c34f5de63b8709c28cf3130ce719a78a5be352b44 not found: ID does not exist"
Oct 30 22:19:12.907195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:12.907147    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podStartSLOduration=2.9071300300000003 podStartE2EDuration="2.90713003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:12.90692083 +0000 UTC m=+2161.890827706" watchObservedRunningTime="2026-10-30 22:19:12.90713003 +0000 UTC m=+2161.891036907"
Oct 30 22:19:13.012286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.012198    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 22:19:13.015910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.015876    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f776cb-6667b5947d-qgstn"]
Oct 30 22:19:13.126253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.126218    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" path="/var/lib/kubelet/pods/a3a70268-c93c-4ef6-ae1c-bae0636c20f5/volumes"
Oct 30 22:19:13.281912 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:13.281835    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:19:13.324279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.324238    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:13.329997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.329970    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:13.757953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.757921    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:13.759084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:13.759063    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:19:14.742723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.742663    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:19:14.767090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.767064    5510 generic.go:356] "Generic (PLEG): container finished" podID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerID="7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511" exitCode=0
Oct 30 22:19:14.767487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.767118    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"
Oct 30 22:19:14.767487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.767144    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerDied","Data":"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"}
Oct 30 22:19:14.767487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.767219    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd" event={"ID":"5d2b331c-a9a6-42b0-93df-8465432bd116","Type":"ContainerDied","Data":"be39796db8576280e7dfe235c18bae998d80adc3c0d7c6db471ad17916855960"}
Oct 30 22:19:14.767487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.767249    5510 scope.go:122] "RemoveContainer" containerID="7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"
Oct 30 22:19:14.777106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.777082    5510 scope.go:122] "RemoveContainer" containerID="5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"
Oct 30 22:19:14.786274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.786257    5510 scope.go:122] "RemoveContainer" containerID="7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"
Oct 30 22:19:14.786550 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:14.786532    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511\": container with ID starting with 7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511 not found: ID does not exist" containerID="7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"
Oct 30 22:19:14.786599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.786558    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511"} err="failed to get container status \"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511\": rpc error: code = NotFound desc = could not find container \"7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511\": container with ID starting with 7a21cfab492ff49c063a8fdafd5a16a2d90240e53bee0d71c696c7e13a110511 not found: ID does not exist"
Oct 30 22:19:14.786599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.786575    5510 scope.go:122] "RemoveContainer" containerID="5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"
Oct 30 22:19:14.786841 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:14.786823    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49\": container with ID starting with 5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49 not found: ID does not exist" containerID="5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"
Oct 30 22:19:14.786905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.786845    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49"} err="failed to get container status \"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49\": rpc error: code = NotFound desc = could not find container \"5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49\": container with ID starting with 5d8459951d3d6ef067fd2084346a6a7cae383ae96fe826dde79da42c1990cf49 not found: ID does not exist"
Oct 30 22:19:14.876359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876333    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876380    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876408    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876446    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876503    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876630    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:14.876679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.876930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876768    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:14.876930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876818    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:14.877071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.876922    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:14.877071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.877010    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.877177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.877148    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.877238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.877181    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\") pod \"5d2b331c-a9a6-42b0-93df-8465432bd116\" (UID: \"5d2b331c-a9a6-42b0-93df-8465432bd116\") "
Oct 30 22:19:14.877600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.877575    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:14.878063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878040    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d2b331c-a9a6-42b0-93df-8465432bd116-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.878175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878068    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.878175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878093    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.878175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878109    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.878175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878125    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.878814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878764    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:14.878898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.878863    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:14.879262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.879244    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "kube-api-access-z7gk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:14.879428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.879412    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token" pod "5d2b331c-a9a6-42b0-93df-8465432bd116" (UID: "5d2b331c-a9a6-42b0-93df-8465432bd116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:14.979328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.979294    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.979328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.979323    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.979328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.979333    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d2b331c-a9a6-42b0-93df-8465432bd116-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.979555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:14.979343    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gk5\" (UniqueName: \"kubernetes.io/projected/5d2b331c-a9a6-42b0-93df-8465432bd116-kube-api-access-z7gk5\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:15.180691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:15.180658    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:19:15.189365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:15.189334    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-c9dc10-openshift-default-79b996974f-b8jpd"]
Oct 30 22:19:15.770753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:15.770721    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:15.917594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:15.917561    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.071179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.071073    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.072774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.072752    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:19:16.073371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073354    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.073371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073370    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.073487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073404    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.073487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073410    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.073487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073423    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing"
Oct 30 22:19:16.073487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073428    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing"
Oct 30 22:19:16.073622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073508    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.073622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073515    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a70268-c93c-4ef6-ae1c-bae0636c20f5" containerName="payload-processing"
Oct 30 22:19:16.073622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.073527    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" containerName="istio-proxy"
Oct 30 22:19:16.078968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.078946    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.173858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.173829    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-778d39d3-serving-cert\""
Oct 30 22:19:16.176818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.176789    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:19:16.196065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.196032    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm7j7\" (UniqueName: \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.196302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.196277    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.196374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.196327    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.196433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.196400    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.298046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.298015    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm7j7\" (UniqueName: \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.298224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.298159    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.298224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.298182    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.298224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.298201    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.298337 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:16.298319    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-serving-cert: secret "maas-api-e2e-mt-778d39d3-serving-cert" not found
Oct 30 22:19:16.298392 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:16.298383    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls podName:a5483f84-4a45-44e0-9255-bd0dfaa44cd9 nodeName:}" failed. No retries permitted until 2026-10-30 22:19:16.79836784 +0000 UTC m=+2165.782274707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls") pod "maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" (UID: "a5483f84-4a45-44e0-9255-bd0dfaa44cd9") : secret "maas-api-e2e-mt-778d39d3-serving-cert" not found
Oct 30 22:19:16.298950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.298927    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.300752 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.300729    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.300918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.300885    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.364675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.364604    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm7j7\" (UniqueName: \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.571698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.571669    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.784241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.784212    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:16.806678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.806644    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.809238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.809212    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") pod \"maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:16.994592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:16.994543    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:17.129581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:17.129547    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2b331c-a9a6-42b0-93df-8465432bd116" path="/var/lib/kubelet/pods/5d2b331c-a9a6-42b0-93df-8465432bd116/volumes"
Oct 30 22:19:17.379316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:17.379235    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:19:17.386280 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:19:17.386248    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5483f84_4a45_44e0_9255_bd0dfaa44cd9.slice/crio-6cee0d3a9ebb52fa8aaa45c8267f9f3be10c19337a60ba64c285a8c87a49a3be WatchSource:0}: Error finding container 6cee0d3a9ebb52fa8aaa45c8267f9f3be10c19337a60ba64c285a8c87a49a3be: Status 404 returned error can't find the container with id 6cee0d3a9ebb52fa8aaa45c8267f9f3be10c19337a60ba64c285a8c87a49a3be
Oct 30 22:19:17.387771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:17.387751    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:19:17.789714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:17.789680    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerStarted","Data":"6cee0d3a9ebb52fa8aaa45c8267f9f3be10c19337a60ba64c285a8c87a49a3be"}
Oct 30 22:19:18.347274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:18.347239    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:18.501595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:18.501514    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:18.800438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:18.800333    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerStarted","Data":"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"}
Oct 30 22:19:18.800911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:18.800437    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:18.913479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:18.913419    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" podStartSLOduration=2.074667712 podStartE2EDuration="2.913403763s" totalImagesPullingTime="838.736051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:18.911093561 +0000 UTC m=+2167.895000429" watchObservedRunningTime="2026-10-30 22:19:18.913403763 +0000 UTC m=+2167.897310660"
Oct 30 22:19:19.471633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:19.471597    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:19.678844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:19.678813    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:19.949667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:19.949582    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.094622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.094585    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:19:20.099629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.099604    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.102700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.102673    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.151372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.151342    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:19:20.211808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.209018    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-edf6d5-dockercfg-rhss4\""
Oct 30 22:19:20.211808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.209170    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-edf6d5\""
Oct 30 22:19:20.277608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.277569    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.278187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.278162    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhmc\" (UniqueName: \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.312123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.312088    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:19:20.316891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.316868    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.336286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.335738    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:19:20.379299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.379267    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhmc\" (UniqueName: \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.379492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.379382    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.380100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.380080    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.434288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.434254    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhmc\" (UniqueName: \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\") pod \"payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:20.481322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.481234    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7zz\" (UniqueName: \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.481490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.481467    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.583802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.582504    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7zz\" (UniqueName: \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.583802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.582723    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.583802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.583684    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.637270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.637241    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7zz\" (UniqueName: \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\") pod \"payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.647009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.646985    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:20.725216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:20.719102    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:21.030240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.030151    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:19:21.036449 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:19:21.036418    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87429923_6520_4079_9c03_88d308793ffa.slice/crio-988510bfc0c882d9ea2783ad87c4ead02454bec340a9b6b7dca5c395c77a3482 WatchSource:0}: Error finding container 988510bfc0c882d9ea2783ad87c4ead02454bec340a9b6b7dca5c395c77a3482: Status 404 returned error can't find the container with id 988510bfc0c882d9ea2783ad87c4ead02454bec340a9b6b7dca5c395c77a3482
Oct 30 22:19:21.134320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.134298    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:19:21.141117 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:19:21.141091    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b326b3f_8697_46f6_ace0_e167448c9a0d.slice/crio-a17ce93d72388312831faca7fe4666dacf2872e7268b4f8c43478f485cc88029 WatchSource:0}: Error finding container a17ce93d72388312831faca7fe4666dacf2872e7268b4f8c43478f485cc88029: Status 404 returned error can't find the container with id a17ce93d72388312831faca7fe4666dacf2872e7268b4f8c43478f485cc88029
Oct 30 22:19:21.456429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.456351    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.608736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.608703    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.872190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.872138    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" event={"ID":"87429923-6520-4079-9c03-88d308793ffa","Type":"ContainerStarted","Data":"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"}
Oct 30 22:19:21.872403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.872250    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:21.872403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.872276    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" event={"ID":"87429923-6520-4079-9c03-88d308793ffa","Type":"ContainerStarted","Data":"988510bfc0c882d9ea2783ad87c4ead02454bec340a9b6b7dca5c395c77a3482"}
Oct 30 22:19:21.874295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.874264    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" event={"ID":"6b326b3f-8697-46f6-ace0-e167448c9a0d","Type":"ContainerStarted","Data":"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"}
Oct 30 22:19:21.874451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.874317    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" event={"ID":"6b326b3f-8697-46f6-ace0-e167448c9a0d","Type":"ContainerStarted","Data":"a17ce93d72388312831faca7fe4666dacf2872e7268b4f8c43478f485cc88029"}
Oct 30 22:19:21.874451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.874348    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:21.978558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:21.978501    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" podStartSLOduration=1.9784868329999998 podStartE2EDuration="1.978486833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:21.977671314 +0000 UTC m=+2170.961578190" watchObservedRunningTime="2026-10-30 22:19:21.978486833 +0000 UTC m=+2170.962393709"
Oct 30 22:19:22.108815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:22.099253    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" podStartSLOduration=2.099229701 podStartE2EDuration="2.099229701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:22.082297777 +0000 UTC m=+2171.066204654" watchObservedRunningTime="2026-10-30 22:19:22.099229701 +0000 UTC m=+2171.083136582"
Oct 30 22:19:23.193196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:23.193166    5510 scope.go:122] "RemoveContainer" containerID="8826c335763254851d43b9b50dfd77ec65a08451daae52ff224a63eef4ee8a20"
Oct 30 22:19:23.203941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:23.203918    5510 scope.go:122] "RemoveContainer" containerID="6014c6f85254931b0ae557a4545146610cc5773028369fe8704f0307200ad6fa"
Oct 30 22:19:24.857295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:24.857253    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.198:8443/health\": dial tcp 10.133.0.198:8443: connect: connection refused"
Oct 30 22:19:25.638869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:25.638832    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:25.812365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:25.812315    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.064663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:26.064609    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.281350 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:26.281315    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:28.489559 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:28.489518    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5483f84_4a45_44e0_9255_bd0dfaa44cd9.slice/crio-conmon-7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:19:28.935431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:28.935399    5510 generic.go:356] "Generic (PLEG): container finished" podID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerID="7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712" exitCode=1
Oct 30 22:19:28.935608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:28.935448    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerDied","Data":"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"}
Oct 30 22:19:28.935845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:28.935823    5510 scope.go:122] "RemoveContainer" containerID="7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"
Oct 30 22:19:29.240229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:29.240201    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:29.383992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:29.383963    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:29.954407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:29.954367    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerStarted","Data":"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"}
Oct 30 22:19:29.954884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:29.954621    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:30.713384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:30.713352    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:32.886071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:32.886041    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:19:32.886576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:32.886124    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:19:35.049595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:35.049563    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:35.192957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:35.192913    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:35.334440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:35.334340    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:35.967642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:35.967611    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:19:39.727553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:39.727520    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:43.280486 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:19:43.280444    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:19:43.604348 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:43.604265    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:44.996297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:44.996265    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.257264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:45.257190    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.412836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:45.412769    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:45.439117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:45.439085    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"]
Oct 30 22:19:45.490434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:45.490397    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1f64a-tg7kp"]
Oct 30 22:19:45.896146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:45.896104    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:47.128651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:47.128609    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade48ccd-0013-4cc7-8cb2-5a113dff32b3" path="/var/lib/kubelet/pods/ade48ccd-0013-4cc7-8cb2-5a113dff32b3/volumes"
Oct 30 22:19:49.573947 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:49.573909    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:49.727946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:49.727916    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.382106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:50.382079    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:51.745831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:51.745801    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:51.900986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:51.900955    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:52.090167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:52.090076    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.159335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:55.159302    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:59.961079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:19:59.961041    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:00.524239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.524205    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:20:00.529922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.529901    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.685391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.685351    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.685732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.685699    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.685905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.685877    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686108    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7544\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686246    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686322    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686443    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686529    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.686622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.686580    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.727402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.727359    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:20:00.787404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787315    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787377    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787415    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787501    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787569    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787650    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787709    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7544\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787746    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.787840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787775    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.788078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.787811    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.788281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.788253    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.788381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.788256    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.788440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.788415    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.788518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.788490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.790219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.790194    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.790457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.790435    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.854158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.854120    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7544\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:00.859061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:00.859030    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4c796\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:01.144346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:01.144267    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:01.580944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:01.580903    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:20:01.587113 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:20:01.587080    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21e5add_7f01_4b1c_a8f1_c3071bc83e2e.slice/crio-c2d29e35b4fb63cf324d6a0a33aa9a52684a7c0be143278f07521054c3d04fc5 WatchSource:0}: Error finding container c2d29e35b4fb63cf324d6a0a33aa9a52684a7c0be143278f07521054c3d04fc5: Status 404 returned error can't find the container with id c2d29e35b4fb63cf324d6a0a33aa9a52684a7c0be143278f07521054c3d04fc5
Oct 30 22:20:01.590270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:01.590231    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:01.590362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:01.590323    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:01.590410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:01.590365    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:02.182385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:02.182346    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" event={"ID":"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e","Type":"ContainerStarted","Data":"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"}
Oct 30 22:20:02.182974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:02.182941    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" event={"ID":"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e","Type":"ContainerStarted","Data":"c2d29e35b4fb63cf324d6a0a33aa9a52684a7c0be143278f07521054c3d04fc5"}
Oct 30 22:20:02.293070 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:02.293006    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podStartSLOduration=2.292990064 podStartE2EDuration="2.292990064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.291800054 +0000 UTC m=+2211.275706926" watchObservedRunningTime="2026-10-30 22:20:02.292990064 +0000 UTC m=+2211.276897192"
Oct 30 22:20:03.145168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:03.145123    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:03.147959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:03.147929    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:03.148105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:03.148002    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:04.145130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:04.145092    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:04.145544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:04.145164    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:05.026917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:05.026880    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:05.145585 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:05.145555    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:05.146084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:05.145609    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:06.144680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:06.144645    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:06.144920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:06.144698    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:07.145516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:07.145482    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:07.145954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:07.145545    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:08.145289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:08.145247    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:08.145473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:08.145330    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:09.145689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:09.145648    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.146134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:09.145717    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:20:10.371717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:10.371607    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:10.373653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:10.373621    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:20:10.556440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:10.556399    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:10.556882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:10.556848    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:10.557872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:10.557852    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:20:13.465304 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:13.465235    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:20:15.640173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.640135    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:20:15.645475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.645455    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692224    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692396    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692477 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692460    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692499    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692519    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692576    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692638    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692660    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftsz\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.692739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.692722    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.734229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.734199    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:20:15.743940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.743910    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:15.817209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817049    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.817209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817196    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817240    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817267    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817304    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817399    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817421    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftsz\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817454    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.818240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.817489    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.819367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.819332    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.819474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.819388    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.819740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.819718    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.820360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.820331    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.820513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.820464    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.823056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.823031    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.823227 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.823209    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.879962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.879930    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.880759 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.880734    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftsz\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\") pod \"maas-default-gateway-openshift-default-694db55c87-ffhxj\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:15.881007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.880984    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:20:15.978110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.978069    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:20:15.986603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:15.986578    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:16.278189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.278107    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.467301 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.467265    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.497320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.497282    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:20:16.501048 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:20:16.501005    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69463924_211a_4117_bb6f_eec4aa88a760.slice/crio-bd7ba8693268665bbfeb7b61e35004b79873ef8fd0137d36c7a43afbac860a97 WatchSource:0}: Error finding container bd7ba8693268665bbfeb7b61e35004b79873ef8fd0137d36c7a43afbac860a97: Status 404 returned error can't find the container with id bd7ba8693268665bbfeb7b61e35004b79873ef8fd0137d36c7a43afbac860a97
Oct 30 22:20:16.507880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.507713    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:16.508723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.508050    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:16.508723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:16.508106    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:17.516806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:17.503903    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" event={"ID":"69463924-211a-4117-bb6f-eec4aa88a760","Type":"ContainerStarted","Data":"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"}
Oct 30 22:20:17.516806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:17.503972    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" event={"ID":"69463924-211a-4117-bb6f-eec4aa88a760","Type":"ContainerStarted","Data":"bd7ba8693268665bbfeb7b61e35004b79873ef8fd0137d36c7a43afbac860a97"}
Oct 30 22:20:17.987253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:17.987216    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:17.988846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:17.988822    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:17.988959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:17.988875    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:18.987806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:18.987477    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:18.987806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:18.987549    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:19.987316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:19.987270    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:19.987519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:19.987347    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:20.882913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:20.882855    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" podUID="ac544671-78b1-4eb9-8cbd-9c4792aad802" containerName="payload-processing" containerID="cri-o://e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c" gracePeriod=25
Oct 30 22:20:20.981883 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:20.979843    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" podUID="f68ea981-4d28-4d8f-bbcc-dddf76667082" containerName="payload-pre-processing" containerID="cri-o://5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc" gracePeriod=25
Oct 30 22:20:20.986921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:20.986889    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:20.987059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:20.986947    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:21.000812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.000709    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:21.340875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.340844    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:20:21.416591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.416562    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:20:21.436046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436021    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\" (UniqueName: \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\") pod \"f68ea981-4d28-4d8f-bbcc-dddf76667082\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") "
Oct 30 22:20:21.436205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436092    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\") pod \"ac544671-78b1-4eb9-8cbd-9c4792aad802\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") "
Oct 30 22:20:21.436205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436145    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\") pod \"f68ea981-4d28-4d8f-bbcc-dddf76667082\" (UID: \"f68ea981-4d28-4d8f-bbcc-dddf76667082\") "
Oct 30 22:20:21.436324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436310    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\" (UniqueName: \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\") pod \"ac544671-78b1-4eb9-8cbd-9c4792aad802\" (UID: \"ac544671-78b1-4eb9-8cbd-9c4792aad802\") "
Oct 30 22:20:21.436545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436504    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume" pod "ac544671-78b1-4eb9-8cbd-9c4792aad802" (UID: "ac544671-78b1-4eb9-8cbd-9c4792aad802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:21.436665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436558    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume" pod "f68ea981-4d28-4d8f-bbcc-dddf76667082" (UID: "f68ea981-4d28-4d8f-bbcc-dddf76667082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:21.437009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.436985    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac544671-78b1-4eb9-8cbd-9c4792aad802-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:21.437103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.437016    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f68ea981-4d28-4d8f-bbcc-dddf76667082-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:21.438318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.438296    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8" pod "f68ea981-4d28-4d8f-bbcc-dddf76667082" (UID: "f68ea981-4d28-4d8f-bbcc-dddf76667082"). InnerVolumeSpecName "kube-api-access-f9hw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:21.438685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.438660    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr" pod "ac544671-78b1-4eb9-8cbd-9c4792aad802" (UID: "ac544671-78b1-4eb9-8cbd-9c4792aad802"). InnerVolumeSpecName "kube-api-access-79bwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:21.493804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.484301    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podStartSLOduration=6.484280571 podStartE2EDuration="6.484280571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:17.639368647 +0000 UTC m=+2226.623275523" watchObservedRunningTime="2026-10-30 22:20:21.484280571 +0000 UTC m=+2230.468187449"
Oct 30 22:20:21.537715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.537684    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bwr\" (UniqueName: \"kubernetes.io/projected/ac544671-78b1-4eb9-8cbd-9c4792aad802-kube-api-access-79bwr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:21.537715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.537712    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hw8\" (UniqueName: \"kubernetes.io/projected/f68ea981-4d28-4d8f-bbcc-dddf76667082-kube-api-access-f9hw8\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:21.538971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.538935    5510 generic.go:356] "Generic (PLEG): container finished" podID="f68ea981-4d28-4d8f-bbcc-dddf76667082" containerID="5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc" exitCode=0
Oct 30 22:20:21.539107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.538993    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"
Oct 30 22:20:21.539107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.539046    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" event={"ID":"f68ea981-4d28-4d8f-bbcc-dddf76667082","Type":"ContainerDied","Data":"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"}
Oct 30 22:20:21.539107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.539099    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f" event={"ID":"f68ea981-4d28-4d8f-bbcc-dddf76667082","Type":"ContainerDied","Data":"10ee38c2451be6f9b243994ec43a6ccec0d0d9dc1be213c23300a7e6719beab8"}
Oct 30 22:20:21.539281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.539133    5510 scope.go:122] "RemoveContainer" containerID="5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"
Oct 30 22:20:21.542031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.541580    5510 generic.go:356] "Generic (PLEG): container finished" podID="ac544671-78b1-4eb9-8cbd-9c4792aad802" containerID="e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c" exitCode=0
Oct 30 22:20:21.542031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.541638    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" event={"ID":"ac544671-78b1-4eb9-8cbd-9c4792aad802","Type":"ContainerDied","Data":"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"}
Oct 30 22:20:21.542031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.541678    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml" event={"ID":"ac544671-78b1-4eb9-8cbd-9c4792aad802","Type":"ContainerDied","Data":"173aed796426e3565b00a20aef255f3fb01724b4de8510883df83e8bed7565cd"}
Oct 30 22:20:21.542031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.541749    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"
Oct 30 22:20:21.586511 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:21.586482    5510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68ea981_4d28_4d8f_bbcc_dddf76667082.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:20:21.587671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.587339    5510 scope.go:122] "RemoveContainer" containerID="5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"
Oct 30 22:20:21.594242 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:21.593685    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc\": container with ID starting with 5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc not found: ID does not exist" containerID="5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"
Oct 30 22:20:21.594242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.593733    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc"} err="failed to get container status \"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc\": rpc error: code = NotFound desc = could not find container \"5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc\": container with ID starting with 5c4b727c228b97bfe73cf1211ba2c606725f8736b382f2f9407158534da17cdc not found: ID does not exist"
Oct 30 22:20:21.594242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.593761    5510 scope.go:122] "RemoveContainer" containerID="e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"
Oct 30 22:20:21.616687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.616629    5510 scope.go:122] "RemoveContainer" containerID="e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"
Oct 30 22:20:21.617101 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:21.617084    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c\": container with ID starting with e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c not found: ID does not exist" containerID="e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"
Oct 30 22:20:21.617207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.617109    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c"} err="failed to get container status \"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c\": rpc error: code = NotFound desc = could not find container \"e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c\": container with ID starting with e132e34bd42ee449b3e88407532a1d886db0af9507df25c7a212adcf507f9c4c not found: ID does not exist"
Oct 30 22:20:21.800212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.800175    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:20:21.811839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.811799    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4215ed-77b6c784c9-hp29f"]
Oct 30 22:20:21.915986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.915953    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:20:21.926320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.926289    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4215ed-c4f54d9d-rgfml"]
Oct 30 22:20:21.987864 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.987826    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:21.988062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:21.987883    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:22.987072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:22.987029    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:22.987514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:22.987102    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:23.127089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.127010    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac544671-78b1-4eb9-8cbd-9c4792aad802" path="/var/lib/kubelet/pods/ac544671-78b1-4eb9-8cbd-9c4792aad802/volumes"
Oct 30 22:20:23.127465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.127448    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68ea981-4d28-4d8f-bbcc-dddf76667082" path="/var/lib/kubelet/pods/f68ea981-4d28-4d8f-bbcc-dddf76667082/volumes"
Oct 30 22:20:23.602847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.602814    5510 generic.go:356] "Generic (PLEG): container finished" podID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerID="cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be" exitCode=0
Oct 30 22:20:23.603075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.602905    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerDied","Data":"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"}
Oct 30 22:20:23.603442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.603421    5510 scope.go:122] "RemoveContainer" containerID="cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"
Oct 30 22:20:23.605251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.605220    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:23.605371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.605307    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:23.605371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.605358    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:23.987078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.987039    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 22:20:23.987472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:23.987109    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 22:20:24.616804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:24.615720    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerStarted","Data":"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"}
Oct 30 22:20:24.994807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:24.994230    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:20:24.997828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:24.996317    5510 patch_prober.go:29] interesting pod/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:20:24.997828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:24.996378    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:20:25.441557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.440805    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:25.441557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.441120    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:25.442223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.442204    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:20:25.790456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.790424    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:25.973806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.959890    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:25.996806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.990464    5510 patch_prober.go:29] interesting pod/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.996806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:25.990554    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:20:26.134907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:26.134831    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:26.934505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:26.934469    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:20:26.989437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:26.989396    5510 patch_prober.go:29] interesting pod/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.989639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:26.989461    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:20:27.990132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:27.990092    5510 patch_prober.go:29] interesting pod/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:20:27.990569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:27.990176    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:20:28.989807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:28.989739    5510 patch_prober.go:29] interesting pod/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:20:28.990042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:28.989838    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:20:29.575337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:29.575291    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:30.037712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:30.037680    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:20:30.039897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:30.039872    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:20:30.205339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:30.205300    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:31.724933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:31.724892    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:31.977384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:31.977296    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.120424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:32.120377    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.222254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:32.222220    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"]
Oct 30 22:20:32.268468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:32.268391    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fef81-6pr6c"]
Oct 30 22:20:32.646558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:32.646477    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:33.127225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:33.127193    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cfb3ee-0703-4440-9bb0-c488f30d2162" path="/var/lib/kubelet/pods/d5cfb3ee-0703-4440-9bb0-c488f30d2162/volumes"
Oct 30 22:20:35.892562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.892520    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:20:35.893412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893387    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f68ea981-4d28-4d8f-bbcc-dddf76667082" containerName="payload-pre-processing"
Oct 30 22:20:35.893532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893413    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68ea981-4d28-4d8f-bbcc-dddf76667082" containerName="payload-pre-processing"
Oct 30 22:20:35.893532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893444    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ac544671-78b1-4eb9-8cbd-9c4792aad802" containerName="payload-processing"
Oct 30 22:20:35.893532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893453    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac544671-78b1-4eb9-8cbd-9c4792aad802" containerName="payload-processing"
Oct 30 22:20:35.893699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893664    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68ea981-4d28-4d8f-bbcc-dddf76667082" containerName="payload-pre-processing"
Oct 30 22:20:35.893699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.893682    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac544671-78b1-4eb9-8cbd-9c4792aad802" containerName="payload-processing"
Oct 30 22:20:35.897449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.897427    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:35.946603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.946580    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-778d39d3-openshift-default-dockercfg-9jkcd\""
Oct 30 22:20:35.996247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:35.996217    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:20:36.011877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.011844    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.011895    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.011969    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012027    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012045    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsmf\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012158    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012188    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012210    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.012394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.012369    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.113981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.113937    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114006    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114043    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114104    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114128    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsmf\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114268    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114304    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114346    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114406    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.114730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114444    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.115053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.114888    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.115182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.115159    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.115501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.115469    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.117035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.116996    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.117279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.117261    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.169265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.169194    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsmf\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.169265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.169226    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.211464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.211436    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:36.642052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.642015    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:20:36.648747 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:20:36.648715    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c989a62_4da2_4eb5_9743_a38629eb116b.slice/crio-b41677b4ed39c030d5d8566644bca6aba463a8f6be42c3742fba6d037de6858d WatchSource:0}: Error finding container b41677b4ed39c030d5d8566644bca6aba463a8f6be42c3742fba6d037de6858d: Status 404 returned error can't find the container with id b41677b4ed39c030d5d8566644bca6aba463a8f6be42c3742fba6d037de6858d
Oct 30 22:20:36.651217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.651183    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:36.651290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.651255    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:36.651290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.651283    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:36.747238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.747201    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" event={"ID":"5c989a62-4da2-4eb5-9743-a38629eb116b","Type":"ContainerStarted","Data":"a5f8f0cca02651d685b9e2af28cf89f8f1af3bf54996b382359a74b1f59ec381"}
Oct 30 22:20:36.747423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.747268    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" event={"ID":"5c989a62-4da2-4eb5-9743-a38629eb116b","Type":"ContainerStarted","Data":"b41677b4ed39c030d5d8566644bca6aba463a8f6be42c3742fba6d037de6858d"}
Oct 30 22:20:36.864370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:36.864314    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podStartSLOduration=1.86430004 podStartE2EDuration="1.86430004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:36.860973114 +0000 UTC m=+2245.844879991" watchObservedRunningTime="2026-10-30 22:20:36.86430004 +0000 UTC m=+2245.848206991"
Oct 30 22:20:37.052057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:37.052025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:37.213166 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:37.213132    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:37.213545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:37.213524    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:37.213710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:37.213687    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:38.212413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:38.212382    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:38.212860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:38.212441    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:38.970806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:38.951940    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:39.124925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.124882    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:39.211960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.211925    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:39.212172 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.211986    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:39.279609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.279519    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:20:39.280100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.279743    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" containerID="cri-o://aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314" gracePeriod=30
Oct 30 22:20:39.288952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:39.288921    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:40.212155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:40.212120    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:40.212381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:40.212181    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:41.212619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:41.212579    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:41.213117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:41.212638    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:42.212638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:42.212608    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:20:42.213125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:42.212657    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:20:42.348594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:42.348551    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:43.216941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:43.216906    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:43.217365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:43.217342    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:43.218033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:43.218011    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:20:43.255663 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:43.255629    5510 patch_prober.go:29] interesting pod/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:20:43.255856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:43.255692    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:20:43.284336 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:43.284306    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:20:44.556336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.556273    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:20:44.577620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.577591    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.577731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.577659    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.577810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.577771    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.577903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.577881    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.577934    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578000    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578040    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:44.578168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578081    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:44.578168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578163    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578225    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\") pod \"df8303af-8bf0-4abd-98e5-38b4384a4110\" (UID: \"df8303af-8bf0-4abd-98e5-38b4384a4110\") "
Oct 30 22:20:44.578386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578270    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:44.578386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578338    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:44.578937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578890    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:44.579038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578933    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.579038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578958    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.579038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578973    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8303af-8bf0-4abd-98e5-38b4384a4110-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.579038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.578989    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.580089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.580066    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:44.580314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.580287    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "kube-api-access-c25lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:44.580654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.580623    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:44.580760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.580684    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo" pod "df8303af-8bf0-4abd-98e5-38b4384a4110" (UID: "df8303af-8bf0-4abd-98e5-38b4384a4110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:20:44.680924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.680892    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.680924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.680925    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.681125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.680940    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.681125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.680954    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c25lf\" (UniqueName: \"kubernetes.io/projected/df8303af-8bf0-4abd-98e5-38b4384a4110-kube-api-access-c25lf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.681125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.680971    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8303af-8bf0-4abd-98e5-38b4384a4110-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:44.825720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.825630    5510 generic.go:356] "Generic (PLEG): container finished" podID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerID="aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314" exitCode=0
Oct 30 22:20:44.825720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.825674    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"
Oct 30 22:20:44.826009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.825715    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerDied","Data":"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"}
Oct 30 22:20:44.826009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.825809    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng" event={"ID":"df8303af-8bf0-4abd-98e5-38b4384a4110","Type":"ContainerDied","Data":"06bf7063d2b336ae3afcc9bc5c2a6b49bb879a93eb2173d2a6a4c7b79fa3516c"}
Oct 30 22:20:44.826009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.825837    5510 scope.go:122] "RemoveContainer" containerID="aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"
Oct 30 22:20:44.837387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.837366    5510 scope.go:122] "RemoveContainer" containerID="f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"
Oct 30 22:20:44.846379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.846361    5510 scope.go:122] "RemoveContainer" containerID="aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"
Oct 30 22:20:44.846649 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:44.846630    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314\": container with ID starting with aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314 not found: ID does not exist" containerID="aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"
Oct 30 22:20:44.846717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.846656    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314"} err="failed to get container status \"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314\": rpc error: code = NotFound desc = could not find container \"aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314\": container with ID starting with aa759bf95cc5ee5b1503e79d48357e698b7c7b8ecd0197891ee316c55bb43314 not found: ID does not exist"
Oct 30 22:20:44.846717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.846673    5510 scope.go:122] "RemoveContainer" containerID="f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"
Oct 30 22:20:44.846899 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:20:44.846884    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875\": container with ID starting with f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875 not found: ID does not exist" containerID="f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"
Oct 30 22:20:44.846959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.846901    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875"} err="failed to get container status \"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875\": rpc error: code = NotFound desc = could not find container \"f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875\": container with ID starting with f2bd1ed0ba8f9227d4211d86770bae2056ebed02585e532c9b88a29e4bbec875 not found: ID does not exist"
Oct 30 22:20:44.985107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.984925    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:20:44.987911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:44.987887    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5fef81-openshift-default-7fc85cdf47-vszng"]
Oct 30 22:20:45.126745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:45.126669    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" path="/var/lib/kubelet/pods/df8303af-8bf0-4abd-98e5-38b4384a4110/volumes"
Oct 30 22:20:47.567827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:47.567765    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.014026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.013939    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:20:51.014645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.014629    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.014699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.014647    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.014807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.014791    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.014807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.014805    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.020260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.020237    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.053603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.053568    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.053761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.053642    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.053761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.053736    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.053916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.053892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxc8l\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.053997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.053980    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.054058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.054034    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.054130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.054115    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.054195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.054181    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.054248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.054233    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.060301 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.060277    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:20:51.060996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.060979    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.061074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.060998    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8303af-8bf0-4abd-98e5-38b4384a4110" containerName="istio-proxy"
Oct 30 22:20:51.064456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.064441    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.112063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.112031    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:20:51.155378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155347    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxc8l\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.155557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155405    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.155557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155526    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.155691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155641    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.155691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155683    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155708    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155744    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155773    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155833    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.155859    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.156117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156098    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156272    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156350    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156402    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156459    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156497    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.156665    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974fx\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.157590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.157547    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.158230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.157836    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.166212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.158384    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.166212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.158634    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.166212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.163823    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.166212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.164650    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.169733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.169381    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:20:51.178541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.178516    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.209399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.209365    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxc8l\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.211849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.211820    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258404    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258493    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258529    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258565    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258613    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974fx\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258666    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258746    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258819    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.258847    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.260647    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.261065    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.261307    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.261438    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.261529    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.263147    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.266493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.263850    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.313248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.313217    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.313449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.313428    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974fx\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\") pod \"e2e-mt-778d39d3-openshift-default-bff74755c-sntpm\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") " pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.341502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.335657    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:51.381210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.381170    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:51.775896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.775865    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:20:51.789271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.789207    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.789399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.789306    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.789399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.789349    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.832411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.832382    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:20:51.839727 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:20:51.839700    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f81844_3014_4484_bd88_a4f9e909db5a.slice/crio-cf8a2e6a02e762fecd2398bc7e0c74dc02131fb071b8e1852e0cc602c25270a4 WatchSource:0}: Error finding container cf8a2e6a02e762fecd2398bc7e0c74dc02131fb071b8e1852e0cc602c25270a4: Status 404 returned error can't find the container with id cf8a2e6a02e762fecd2398bc7e0c74dc02131fb071b8e1852e0cc602c25270a4
Oct 30 22:20:51.848444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.846446    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.848444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.846599    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.848444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.846690    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:20:51.871714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.871641    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.881502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.881451    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" event={"ID":"e5aa62ce-a010-424b-8957-9d42889a0793","Type":"ContainerStarted","Data":"ab5d390422da8f5c910119724819e4ff0aa872816910622d92052d95a6ad602d"}
Oct 30 22:20:51.881637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.881554    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" event={"ID":"e5aa62ce-a010-424b-8957-9d42889a0793","Type":"ContainerStarted","Data":"9971cf7a5c81a1433213a5d788eb6188bac4285a3172f1f7a955e4c595d94db7"}
Oct 30 22:20:51.883364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:51.883340    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" event={"ID":"e1f81844-3014-4484-bd88-a4f9e909db5a","Type":"ContainerStarted","Data":"cf8a2e6a02e762fecd2398bc7e0c74dc02131fb071b8e1852e0cc602c25270a4"}
Oct 30 22:20:52.044927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.043179    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.048003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.047372    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podStartSLOduration=2.047353405 podStartE2EDuration="2.047353405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:52.002974306 +0000 UTC m=+2260.986881183" watchObservedRunningTime="2026-10-30 22:20:52.047353405 +0000 UTC m=+2261.031260281"
Oct 30 22:20:52.336655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.336569    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:52.338057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.338030    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:52.338158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.338089    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:52.776806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.776689    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.906487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:52.906458    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" event={"ID":"e1f81844-3014-4484-bd88-a4f9e909db5a","Type":"ContainerStarted","Data":"1a4536d2c366f759e8060b81bfe03b118805ab39bef015629400ca4ae5a2c1b1"}
Oct 30 22:20:53.050948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.038980    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podStartSLOduration=3.038960216 podStartE2EDuration="3.038960216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:53.037725165 +0000 UTC m=+2262.021632052" watchObservedRunningTime="2026-10-30 22:20:53.038960216 +0000 UTC m=+2262.022867094"
Oct 30 22:20:53.339074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.338991    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:53.339321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.339295    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:53.399133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.399093    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:20:53.435012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.418351    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:53.435012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:53.418410    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:54.336634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:54.336601    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:54.337042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:54.336653    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:54.395804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:54.394898    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:54.395804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:54.394961    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:55.337249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:55.337215    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.337677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:55.337269    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:55.390803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:55.390035    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:55.390803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:55.390096    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:56.336902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:56.336868    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.337122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:56.336918    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:56.382805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:56.382226    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:56.382805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:56.382303    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:57.363848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.350823    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:20:57.363848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.362976    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.363848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.363044    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:57.388565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.388527    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:57.389176 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.389141    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:57.837493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:57.837442    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:58.114381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.114289    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:58.280325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.280233    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:58.346938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.346899    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.347489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.347453    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 22:20:58.384474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.384385    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:58.384758 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:58.384725    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:59.386858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:59.385616    5510 patch_prober.go:29] interesting pod/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 30 22:20:59.386858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:59.385681    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 30 22:20:59.632334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:59.632301    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:59.633869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:59.633846    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:20:59.635297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:20:59.635279    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:21:00.681805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:00.676595    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:21:00.681805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:00.677941    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:21:00.683589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:00.683335    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:21:01.272311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:01.272271    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:01.308716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:01.308684    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 22:21:01.308904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:01.308887    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" containerID="cri-o://412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee" gracePeriod=30
Oct 30 22:21:01.609716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:01.609617    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.046813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:02.046752    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.351389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:02.351305    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" podUID="e9162831-1468-4689-a5df-3942674ebfdc" containerName="payload-processing" containerID="cri-o://83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a" gracePeriod=25
Oct 30 22:21:02.820806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:02.795801    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:02.906868 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:02.906799    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:21:03.122813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.115463    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\") pod \"e9162831-1468-4689-a5df-3942674ebfdc\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") "
Oct 30 22:21:03.122813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.115543    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\" (UniqueName: \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\") pod \"e9162831-1468-4689-a5df-3942674ebfdc\" (UID: \"e9162831-1468-4689-a5df-3942674ebfdc\") "
Oct 30 22:21:03.122813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.120410    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume" pod "e9162831-1468-4689-a5df-3942674ebfdc" (UID: "e9162831-1468-4689-a5df-3942674ebfdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:03.168818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.168712    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9162831-1468-4689-a5df-3942674ebfdc-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.302327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.302287    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s" pod "e9162831-1468-4689-a5df-3942674ebfdc" (UID: "e9162831-1468-4689-a5df-3942674ebfdc"). InnerVolumeSpecName "kube-api-access-g7f5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:03.337631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.337593    5510 generic.go:356] "Generic (PLEG): container finished" podID="e9162831-1468-4689-a5df-3942674ebfdc" containerID="83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a" exitCode=0
Oct 30 22:21:03.342821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.338009    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" event={"ID":"e9162831-1468-4689-a5df-3942674ebfdc","Type":"ContainerDied","Data":"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"}
Oct 30 22:21:03.342821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.338073    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh" event={"ID":"e9162831-1468-4689-a5df-3942674ebfdc","Type":"ContainerDied","Data":"ae9eda6254885afeab1288735c88c3c242ab724385c089557f8b0445b96d8301"}
Oct 30 22:21:03.342821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.338100    5510 scope.go:122] "RemoveContainer" containerID="83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"
Oct 30 22:21:03.342821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.338266    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"
Oct 30 22:21:03.376805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.373693    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7f5s\" (UniqueName: \"kubernetes.io/projected/e9162831-1468-4689-a5df-3942674ebfdc-kube-api-access-g7f5s\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.401415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.401388    5510 scope.go:122] "RemoveContainer" containerID="83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"
Oct 30 22:21:03.401705 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:03.401680    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a\": container with ID starting with 83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a not found: ID does not exist" containerID="83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"
Oct 30 22:21:03.401763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.401715    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a"} err="failed to get container status \"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a\": rpc error: code = NotFound desc = could not find container \"83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a\": container with ID starting with 83302cf41310ff1b3660b81f5459e5fefddcbd60598c3cd9eb173bca5721120a not found: ID does not exist"
Oct 30 22:21:03.503273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.503237    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:21:03.507444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:03.507417    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-778d39d3-5cc665b758-775sh"]
Oct 30 22:21:04.390672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.390636    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.558801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.552440    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.704392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.704303    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.897125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.897091    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.940840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.940800    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"]
Oct 30 22:21:04.987088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:04.987056    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f776cb-tz44t"]
Oct 30 22:21:05.126116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:05.126079    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f" path="/var/lib/kubelet/pods/6bd7094d-d0c7-4d6f-9d40-89ffb9e8ac0f/volumes"
Oct 30 22:21:05.126659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:05.126639    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9162831-1468-4689-a5df-3942674ebfdc" path="/var/lib/kubelet/pods/e9162831-1468-4689-a5df-3942674ebfdc/volumes"
Oct 30 22:21:05.373467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:05.373358    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:05.790917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:05.790879    5510 patch_prober.go:29] interesting pod/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:21:05.791355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:05.790958    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:06.597222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.597150    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 22:21:06.659909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.659863    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.659909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.659910    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.659957    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660128    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660174    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660227    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660273    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660301    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660379    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\") pod \"7eefeb76-ce93-44d6-a904-c60c75b6861c\" (UID: \"7eefeb76-ce93-44d6-a904-c60c75b6861c\") "
Oct 30 22:21:06.660576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660545    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:06.660691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660663    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:06.660814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660758    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:06.661012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.660980    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:06.661136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.661113    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:06.661203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.661148    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.661203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.661169    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.661203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.661183    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.661203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.661200    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.662724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.662661    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:06.663212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.663179    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:06.663334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.663310    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:06.663715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.663694    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f" pod "7eefeb76-ce93-44d6-a904-c60c75b6861c" (UID: "7eefeb76-ce93-44d6-a904-c60c75b6861c"). InnerVolumeSpecName "kube-api-access-nqv8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:06.762599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.762544    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqv8f\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-kube-api-access-nqv8f\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.762599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.762583    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.762599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.762593    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.762599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.762603    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefeb76-ce93-44d6-a904-c60c75b6861c-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:06.762992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:06.762612    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefeb76-ce93-44d6-a904-c60c75b6861c-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:07.373070 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.373030    5510 generic.go:356] "Generic (PLEG): container finished" podID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerID="412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee" exitCode=0
Oct 30 22:21:07.373581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.373108    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerDied","Data":"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"}
Oct 30 22:21:07.373581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.373159    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck" event={"ID":"7eefeb76-ce93-44d6-a904-c60c75b6861c","Type":"ContainerDied","Data":"1138672cbd3620c8e6b4d2b7a691b15ce36e516b1dfdec66c0a9378b0b1c5042"}
Oct 30 22:21:07.373581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.373112    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"
Oct 30 22:21:07.373581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.373192    5510 scope.go:122] "RemoveContainer" containerID="412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"
Oct 30 22:21:07.389515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.389494    5510 scope.go:122] "RemoveContainer" containerID="7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"
Oct 30 22:21:07.400023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.399996    5510 scope.go:122] "RemoveContainer" containerID="412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"
Oct 30 22:21:07.400349 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:07.400322    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee\": container with ID starting with 412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee not found: ID does not exist" containerID="412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"
Oct 30 22:21:07.400487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.400353    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee"} err="failed to get container status \"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee\": rpc error: code = NotFound desc = could not find container \"412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee\": container with ID starting with 412d2a27fdbcb8009aac347a277bce9f5ea73556f676ccbe37f270a01b62a4ee not found: ID does not exist"
Oct 30 22:21:07.400487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.400375    5510 scope.go:122] "RemoveContainer" containerID="7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"
Oct 30 22:21:07.400655 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:07.400637    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e\": container with ID starting with 7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e not found: ID does not exist" containerID="7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"
Oct 30 22:21:07.400713 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.400661    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e"} err="failed to get container status \"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e\": rpc error: code = NotFound desc = could not find container \"7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e\": container with ID starting with 7ead52e9888eac44a7684c529f8596c6bf5a31946c0cf22b37ff9bcbbb55503e not found: ID does not exist"
Oct 30 22:21:07.506628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.506567    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:07.532369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.532332    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 22:21:07.577965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:07.577919    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-14af5c-openshift-default-7c9d5d5d57-k9xck"]
Oct 30 22:21:08.859091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:08.859049    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:09.049488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:09.049458    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:09.133502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:09.133417    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" path="/var/lib/kubelet/pods/7eefeb76-ce93-44d6-a904-c60c75b6861c/volumes"
Oct 30 22:21:09.206205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:09.206171    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:10.420850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.420813    5510 generic.go:356] "Generic (PLEG): container finished" podID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerID="4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002" exitCode=0
Oct 30 22:21:10.421298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.420858    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerDied","Data":"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"}
Oct 30 22:21:10.421343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.421320    5510 scope.go:122] "RemoveContainer" containerID="4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"
Oct 30 22:21:10.422975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.422946    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:10.423075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.423016    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:10.423075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:10.423069    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:21:11.189483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:11.189444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:11.429491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:11.429454    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerStarted","Data":"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"}
Oct 30 22:21:11.647357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:11.647325    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:11.805743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:11.805696    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:11.998853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:11.998816    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.158275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.158240    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.334266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.334118    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:21:12.335918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.335892    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:21:12.336214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.336187    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:21:12.362862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.362828    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.572807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:12.572231    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.308730 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:13.308687    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:21:13.324513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:13.324484    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:21:13.324683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:13.324535    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:21:13.766806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:13.762870    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:21:14.324219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:14.324180    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:21:14.324519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:14.324490    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:21:15.323916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:15.323882    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:21:15.324316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:15.323951    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:21:16.324733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:16.324691    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:21:16.325250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:16.324747    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:21:16.697266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:16.697183    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:17.329438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:17.329407    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:21:17.331260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:17.331235    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:21:21.971897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:21.971862    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.809978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:22.809935    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.412367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:26.412331    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:26.581296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:26.581263    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:27.089757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:27.089726    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:29.268031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.267992    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:29.426393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.426364    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:29.614102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.614015    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:21:29.614288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.614262    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" containerID="cri-o://e4ea0946269544fb73e3d6ba5b2a500b1858dcce7dd0fc224fb93ad0207509b4" gracePeriod=30
Oct 30 22:21:29.633247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.633215    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:29.934912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:29.934841    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.216017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:32.215978    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:32.422132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:32.422092    5510 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:21:32.422302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:32.422160    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:21:34.437194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.437156    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:21:34.753515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.753482    5510 generic.go:356] "Generic (PLEG): container finished" podID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerID="e4ea0946269544fb73e3d6ba5b2a500b1858dcce7dd0fc224fb93ad0207509b4" exitCode=0
Oct 30 22:21:34.753664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.753514    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerDied","Data":"e4ea0946269544fb73e3d6ba5b2a500b1858dcce7dd0fc224fb93ad0207509b4"}
Oct 30 22:21:34.753664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.753581    5510 scope.go:122] "RemoveContainer" containerID="e85ce77043b8aff34969045c94cbc522c79b126471611190a18616ffe9fe6ed4"
Oct 30 22:21:34.880083 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.880058    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:21:34.940708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940683    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.940893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940730    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.940893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940750    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.940893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940877    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940921    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940940    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.940990    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:34.941081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941112    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941197    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\") pod \"f42b7201-caa0-4e78-b1ef-c7f89630891b\" (UID: \"f42b7201-caa0-4e78-b1ef-c7f89630891b\") "
Oct 30 22:21:34.941312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941207    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:34.941493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941368    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:34.941593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941572    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:34.941671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941577    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:34.941836 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941815    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.941918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941841    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42b7201-caa0-4e78-b1ef-c7f89630891b-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.941918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941856    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.941918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941867    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.941918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.941875    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:34.943148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.943123    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:34.943248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.943231    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:34.943440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.943414    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:34.943537 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:34.943452    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf" pod "f42b7201-caa0-4e78-b1ef-c7f89630891b" (UID: "f42b7201-caa0-4e78-b1ef-c7f89630891b"). InnerVolumeSpecName "kube-api-access-gdwsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:35.043714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.043628    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwsf\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-kube-api-access-gdwsf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:35.043714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.043669    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:35.043714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.043679    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:35.043714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.043688    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42b7201-caa0-4e78-b1ef-c7f89630891b-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:35.762573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.762533    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk" event={"ID":"f42b7201-caa0-4e78-b1ef-c7f89630891b","Type":"ContainerDied","Data":"25ae67c8f41ed1f5afe211e3d08f17a4edeff38a748945f476e11637f79b7a6c"}
Oct 30 22:21:35.763216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.762596    5510 scope.go:122] "RemoveContainer" containerID="e4ea0946269544fb73e3d6ba5b2a500b1858dcce7dd0fc224fb93ad0207509b4"
Oct 30 22:21:35.763216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.762548    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"
Oct 30 22:21:35.977940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:35.977897    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:21:36.016639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:36.016542    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4215ed-openshift-default-7c76bf4889-wrbtk"]
Oct 30 22:21:37.139560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:37.139527    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" path="/var/lib/kubelet/pods/f42b7201-caa0-4e78-b1ef-c7f89630891b/volumes"
Oct 30 22:21:37.417066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:37.416978    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:41.833560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:41.833525    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:21:41.955958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:41.955924    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:21:42.197146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:42.197068    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:42.366818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:42.366736    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.437361 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:43.437321    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:21:46.403802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.394730    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:21:46.403802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.395040    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api" containerID="cri-o://cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2" gracePeriod=30
Oct 30 22:21:46.514882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.500056    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:46.841583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.840812    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" podUID="b93e1346-4d17-4088-a623-fed64392a217" containerName="payload-processing" containerID="cri-o://aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627" gracePeriod=25
Oct 30 22:21:46.851269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.851086    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:21:46.901852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.901191    5510 generic.go:356] "Generic (PLEG): container finished" podID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerID="cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2" exitCode=0
Oct 30 22:21:46.901852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.901385    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerDied","Data":"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"}
Oct 30 22:21:46.901852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.901443    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7" event={"ID":"a5483f84-4a45-44e0-9255-bd0dfaa44cd9","Type":"ContainerDied","Data":"6cee0d3a9ebb52fa8aaa45c8267f9f3be10c19337a60ba64c285a8c87a49a3be"}
Oct 30 22:21:46.901852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.901476    5510 scope.go:122] "RemoveContainer" containerID="cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"
Oct 30 22:21:46.901852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.901657    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"
Oct 30 22:21:46.956824 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.956576    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" podUID="eea08086-7b2c-4154-bee9-fc576bb6315d" containerName="payload-pre-processing" containerID="cri-o://961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9" gracePeriod=25
Oct 30 22:21:46.975633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:46.975592    5510 scope.go:122] "RemoveContainer" containerID="7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"
Oct 30 22:21:47.028886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.028406    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\") pod \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") "
Oct 30 22:21:47.028886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.028452    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") pod \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") "
Oct 30 22:21:47.028886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.028519    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\" (UniqueName: \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\") pod \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") "
Oct 30 22:21:47.028886 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.028573    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\") pod \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\" (UID: \"a5483f84-4a45-44e0-9255-bd0dfaa44cd9\") "
Oct 30 22:21:47.030009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.029980    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca" pod "a5483f84-4a45-44e0-9255-bd0dfaa44cd9" (UID: "a5483f84-4a45-44e0-9255-bd0dfaa44cd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:47.031662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.031636    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs" pod "a5483f84-4a45-44e0-9255-bd0dfaa44cd9" (UID: "a5483f84-4a45-44e0-9255-bd0dfaa44cd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:47.031811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.031662    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls" pod "a5483f84-4a45-44e0-9255-bd0dfaa44cd9" (UID: "a5483f84-4a45-44e0-9255-bd0dfaa44cd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:47.032978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.032958    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7" pod "a5483f84-4a45-44e0-9255-bd0dfaa44cd9" (UID: "a5483f84-4a45-44e0-9255-bd0dfaa44cd9"). InnerVolumeSpecName "kube-api-access-vm7j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:47.092650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.092621    5510 scope.go:122] "RemoveContainer" containerID="cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"
Oct 30 22:21:47.093145 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:47.093117    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2\": container with ID starting with cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2 not found: ID does not exist" containerID="cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"
Oct 30 22:21:47.093304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.093157    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2"} err="failed to get container status \"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2\": rpc error: code = NotFound desc = could not find container \"cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2\": container with ID starting with cf735eb681988b9289695d28f3de3e3dd0a6b9e24c9fc60e2ccac739cdcba2c2 not found: ID does not exist"
Oct 30 22:21:47.093304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.093185    5510 scope.go:122] "RemoveContainer" containerID="7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"
Oct 30 22:21:47.093542 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:47.093515    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712\": container with ID starting with 7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712 not found: ID does not exist" containerID="7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"
Oct 30 22:21:47.093671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.093549    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712"} err="failed to get container status \"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712\": rpc error: code = NotFound desc = could not find container \"7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712\": container with ID starting with 7f352b6705100a1f5732f538f027a1e3e2d1220b27c3c2f8b2750d79ee695712 not found: ID does not exist"
Oct 30 22:21:47.133754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.133717    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.133754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.133753    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.134027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.133773    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm7j7\" (UniqueName: \"kubernetes.io/projected/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-kube-api-access-vm7j7\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.134027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.133807    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5483f84-4a45-44e0-9255-bd0dfaa44cd9-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.153686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.153155    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:21:47.234361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.234327    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\") pod \"b93e1346-4d17-4088-a623-fed64392a217\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") "
Oct 30 22:21:47.234556 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.234484    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\" (UniqueName: \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\") pod \"b93e1346-4d17-4088-a623-fed64392a217\" (UID: \"b93e1346-4d17-4088-a623-fed64392a217\") "
Oct 30 22:21:47.235059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.235032    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume" pod "b93e1346-4d17-4088-a623-fed64392a217" (UID: "b93e1346-4d17-4088-a623-fed64392a217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:47.235335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.235313    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93e1346-4d17-4088-a623-fed64392a217-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.237301 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.237276    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44" pod "b93e1346-4d17-4088-a623-fed64392a217" (UID: "b93e1346-4d17-4088-a623-fed64392a217"). InnerVolumeSpecName "kube-api-access-h9t44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:47.272805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.272214    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:21:47.345850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.338040    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9t44\" (UniqueName: \"kubernetes.io/projected/b93e1346-4d17-4088-a623-fed64392a217-kube-api-access-h9t44\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.503818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.448610    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\") pod \"eea08086-7b2c-4154-bee9-fc576bb6315d\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") "
Oct 30 22:21:47.503818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.448702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\" (UniqueName: \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\") pod \"eea08086-7b2c-4154-bee9-fc576bb6315d\" (UID: \"eea08086-7b2c-4154-bee9-fc576bb6315d\") "
Oct 30 22:21:47.503818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.452921    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:21:47.503818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.453363    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume" pod "eea08086-7b2c-4154-bee9-fc576bb6315d" (UID: "eea08086-7b2c-4154-bee9-fc576bb6315d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:47.503818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.490327    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj" pod "eea08086-7b2c-4154-bee9-fc576bb6315d" (UID: "eea08086-7b2c-4154-bee9-fc576bb6315d"). InnerVolumeSpecName "kube-api-access-hd5dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:47.516253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.516207    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778d39d3-75856cfcc7-dzzl7"]
Oct 30 22:21:47.551002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.550974    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea08086-7b2c-4154-bee9-fc576bb6315d-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.551002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.550999    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5dj\" (UniqueName: \"kubernetes.io/projected/eea08086-7b2c-4154-bee9-fc576bb6315d-kube-api-access-hd5dj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:47.909697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.909665    5510 generic.go:356] "Generic (PLEG): container finished" podID="eea08086-7b2c-4154-bee9-fc576bb6315d" containerID="961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9" exitCode=0
Oct 30 22:21:47.909911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.909735    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"
Oct 30 22:21:47.909911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.909756    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" event={"ID":"eea08086-7b2c-4154-bee9-fc576bb6315d","Type":"ContainerDied","Data":"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"}
Oct 30 22:21:47.909911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.909839    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh" event={"ID":"eea08086-7b2c-4154-bee9-fc576bb6315d","Type":"ContainerDied","Data":"8dc3971b96dad66c3bf1606b37e32442519a11e8bd346ac22f8f5096ffb598c6"}
Oct 30 22:21:47.909911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.909877    5510 scope.go:122] "RemoveContainer" containerID="961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"
Oct 30 22:21:47.912480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.912458    5510 generic.go:356] "Generic (PLEG): container finished" podID="b93e1346-4d17-4088-a623-fed64392a217" containerID="aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627" exitCode=0
Oct 30 22:21:47.912622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.912510    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"
Oct 30 22:21:47.912622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.912567    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" event={"ID":"b93e1346-4d17-4088-a623-fed64392a217","Type":"ContainerDied","Data":"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"}
Oct 30 22:21:47.912703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.912624    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh" event={"ID":"b93e1346-4d17-4088-a623-fed64392a217","Type":"ContainerDied","Data":"645bee064548bc0bf3fa07d3d2e2321575a72d70fcee60811b7e597bb5494bbb"}
Oct 30 22:21:47.919982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.919963    5510 scope.go:122] "RemoveContainer" containerID="961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"
Oct 30 22:21:47.920287 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:47.920260    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9\": container with ID starting with 961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9 not found: ID does not exist" containerID="961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"
Oct 30 22:21:47.920392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.920296    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9"} err="failed to get container status \"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9\": rpc error: code = NotFound desc = could not find container \"961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9\": container with ID starting with 961e6ba1d4a69b81981552512526c7d0318ff1af22026cbf37e428768a0aedc9 not found: ID does not exist"
Oct 30 22:21:47.920392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.920319    5510 scope.go:122] "RemoveContainer" containerID="aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"
Oct 30 22:21:47.929517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.929496    5510 scope.go:122] "RemoveContainer" containerID="aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"
Oct 30 22:21:47.929751 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:21:47.929734    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627\": container with ID starting with aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627 not found: ID does not exist" containerID="aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"
Oct 30 22:21:47.929816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:47.929757    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627"} err="failed to get container status \"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627\": rpc error: code = NotFound desc = could not find container \"aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627\": container with ID starting with aed233224378019edb01408b2e23b06df6125df070f99beff49d98a272b6c627 not found: ID does not exist"
Oct 30 22:21:48.067123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.066866    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:21:48.072363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.071405    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e66edb-7655ddbb6c-46hwh"]
Oct 30 22:21:48.090153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.090122    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:48.174465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.173527    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:21:48.177748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.177696    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e66edb-868f645df4-wqnkh"]
Oct 30 22:21:48.433469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.432110    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:48.662856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:48.662823    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:49.094959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:49.094917    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:49.125326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:49.125297    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" path="/var/lib/kubelet/pods/a5483f84-4a45-44e0-9255-bd0dfaa44cd9/volumes"
Oct 30 22:21:49.125933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:49.125904    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93e1346-4d17-4088-a623-fed64392a217" path="/var/lib/kubelet/pods/b93e1346-4d17-4088-a623-fed64392a217/volumes"
Oct 30 22:21:49.126568 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:49.126528    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea08086-7b2c-4154-bee9-fc576bb6315d" path="/var/lib/kubelet/pods/eea08086-7b2c-4154-bee9-fc576bb6315d/volumes"
Oct 30 22:21:49.302076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:49.302028    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:53.715242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:53.712025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:54.004805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:54.004719    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:21:54.107966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:54.107926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:58.567896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:58.567855    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:59.960091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:21:59.960045    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.585712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.585670    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:22:00.595011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.594977    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:00.595264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595245    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:00.595372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595359    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.595467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595455    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.595562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595551    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.595644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595633    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.595740 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595730    5510 cpu_manager.go:414] "containerMap: removing container" podUID="eea08086-7b2c-4154-bee9-fc576bb6315d" containerName="payload-pre-processing"
Oct 30 22:22:00.595849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595835    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea08086-7b2c-4154-bee9-fc576bb6315d" containerName="payload-pre-processing"
Oct 30 22:22:00.595958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.595947    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:00.596045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596034    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:00.596180 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596168    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:00.596266 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596255    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:00.596359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596348    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b93e1346-4d17-4088-a623-fed64392a217" containerName="payload-processing"
Oct 30 22:22:00.596440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596429    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93e1346-4d17-4088-a623-fed64392a217" containerName="payload-processing"
Oct 30 22:22:00.596545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596534    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e9162831-1468-4689-a5df-3942674ebfdc" containerName="payload-processing"
Oct 30 22:22:00.596648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596636    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9162831-1468-4689-a5df-3942674ebfdc" containerName="payload-processing"
Oct 30 22:22:00.609807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.596900    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:00.610115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.610074    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:00.611444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.611420    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93e1346-4d17-4088-a623-fed64392a217" containerName="payload-processing"
Oct 30 22:22:00.611629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.611616    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9162831-1468-4689-a5df-3942674ebfdc" containerName="payload-processing"
Oct 30 22:22:00.611743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.611731    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.611874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.611854    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5483f84-4a45-44e0-9255-bd0dfaa44cd9" containerName="maas-api"
Oct 30 22:22:00.612022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.612012    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea08086-7b2c-4154-bee9-fc576bb6315d" containerName="payload-pre-processing"
Oct 30 22:22:00.621171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.621147    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.707294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.704588    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.721521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.708349    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv9p\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.722601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.722567    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.722911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.722887    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.723082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.723066    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.723213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.723196    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.723376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.723360    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.726085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.723553    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.726408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.726385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.731052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.721125    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.731298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.728980    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c1e2ab-openshift-default-dockercfg-rh8s2\""
Oct 30 22:22:00.732461 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.732427    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:22:00.827715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827668    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.827941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827759    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.827941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827818    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.827941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827852    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.827941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.828196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.827978    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.828196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.828103    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.828196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.828134    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.828196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.828171    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv9p\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.845712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.845623    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.853222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.853108    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.853480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.853456    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.853814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.853720    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.866194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.866160    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.876877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.874565    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.876877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.876681    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.883189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.883159    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv9p\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.919287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.919249    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\") pod \"e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") " pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:00.943762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.943732    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.999146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:00.992640    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:01.457137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:01.457105    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:22:01.464669 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:22:01.464637    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a26591_e9d7_4d21_8f4f_431a79306b87.slice/crio-042e5d79bb4c3ea86fc383613b5f575154a0919989ec7a75d15b11463dfcc79b WatchSource:0}: Error finding container 042e5d79bb4c3ea86fc383613b5f575154a0919989ec7a75d15b11463dfcc79b: Status 404 returned error can't find the container with id 042e5d79bb4c3ea86fc383613b5f575154a0919989ec7a75d15b11463dfcc79b
Oct 30 22:22:01.467228 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:01.467198    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:22:01.467321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:01.467265    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:22:01.467321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:01.467292    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:22:02.016880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:02.016843    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerStarted","Data":"ef988c80068dd083c8ff21f475d59b2a0e9d6df4f296df636a4ac51d5cd0f45f"}
Oct 30 22:22:02.017283 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:02.016893    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerStarted","Data":"042e5d79bb4c3ea86fc383613b5f575154a0919989ec7a75d15b11463dfcc79b"}
Oct 30 22:22:02.131426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:02.131372    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podStartSLOduration=2.131358414 podStartE2EDuration="2.131358414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:02.129678947 +0000 UTC m=+2331.113585820" watchObservedRunningTime="2026-10-30 22:22:02.131358414 +0000 UTC m=+2331.115265347"
Oct 30 22:22:02.995970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:02.995937    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:03.001909 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:03.001877    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:03.024435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:03.024406    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:03.025232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:03.025213    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:22:03.491256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:03.491226    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:05.064260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:05.064229    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:05.216418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:05.216386    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:05.376464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:05.376378    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.158068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.158032    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.351816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.350881    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.619941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.619905    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.735084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.735045    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:22:08.736401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.736370    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:08.736594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.736578    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:08.737010 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.736990    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eefeb76-ce93-44d6-a904-c60c75b6861c" containerName="istio-proxy"
Oct 30 22:22:08.737159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.737146    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42b7201-caa0-4e78-b1ef-c7f89630891b" containerName="istio-proxy"
Oct 30 22:22:08.741434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.741410    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.821681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.813443    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:22:08.821994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.814231    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5456n\" (UniqueName: \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.826582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.826551    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.842810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.837058    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1e2ab-dockercfg-fwtvx\""
Oct 30 22:22:08.843280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.843255    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1e2ab\""
Oct 30 22:22:08.928773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.928685    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5456n\" (UniqueName: \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.928773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.928764    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.929515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.929485    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:08.982911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:08.982874    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5456n\" (UniqueName: \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\") pod \"payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") " pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:09.025397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.025332    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" podUID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" containerName="main" containerID="cri-o://32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f" gracePeriod=45
Oct 30 22:22:09.128090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.128057    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:09.352598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.352568    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:22:09.437297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437266    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437307    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437419    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437472    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437556    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437621    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:09.437679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437670    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\" (UniqueName: \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\") pod \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\" (UID: \"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e\") "
Oct 30 22:22:09.437856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437682    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:09.437913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.437883    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:09.438302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.438279    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.438375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.438309    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.438375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.438326    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.439683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.439659    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:09.439827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.439720    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:09.439827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.439725    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5" pod "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" (UID: "41a4ee57-fafc-4e90-bfba-2e1811ae0f5e"). InnerVolumeSpecName "kube-api-access-sqhn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:09.516580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.516544    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:22:09.524078 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:22:09.524048    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6f3fb4_d028_4c27_a5c3_5871fe202278.slice/crio-4c9fbcac6368b743d24ddee44dcd43fc627fb88431a8eea724b75e658cc1ef63 WatchSource:0}: Error finding container 4c9fbcac6368b743d24ddee44dcd43fc627fb88431a8eea724b75e658cc1ef63: Status 404 returned error can't find the container with id 4c9fbcac6368b743d24ddee44dcd43fc627fb88431a8eea724b75e658cc1ef63
Oct 30 22:22:09.540044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.540022    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.540139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.540046    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhn5\" (UniqueName: \"kubernetes.io/projected/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-kube-api-access-sqhn5\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:09.540139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:09.540057    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:10.075668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.075634    5510 generic.go:356] "Generic (PLEG): container finished" podID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" containerID="32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f" exitCode=0
Oct 30 22:22:10.075896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.075675    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"
Oct 30 22:22:10.075896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.075715    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" event={"ID":"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e","Type":"ContainerDied","Data":"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"}
Oct 30 22:22:10.075896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.075755    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5" event={"ID":"41a4ee57-fafc-4e90-bfba-2e1811ae0f5e","Type":"ContainerDied","Data":"f8cc5f6f00728c1dab0a2288f25d871f2125796267aebb94d5048beb59cd5352"}
Oct 30 22:22:10.075896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.075776    5510 scope.go:122] "RemoveContainer" containerID="32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"
Oct 30 22:22:10.077867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.077839    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" event={"ID":"6e6f3fb4-d028-4c27-a5c3-5871fe202278","Type":"ContainerStarted","Data":"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"}
Oct 30 22:22:10.078003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.077904    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" event={"ID":"6e6f3fb4-d028-4c27-a5c3-5871fe202278","Type":"ContainerStarted","Data":"4c9fbcac6368b743d24ddee44dcd43fc627fb88431a8eea724b75e658cc1ef63"}
Oct 30 22:22:10.078003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.077936    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:10.087592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.087576    5510 scope.go:122] "RemoveContainer" containerID="32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"
Oct 30 22:22:10.087907 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:10.087885    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f\": container with ID starting with 32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f not found: ID does not exist" containerID="32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"
Oct 30 22:22:10.087996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.087914    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f"} err="failed to get container status \"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f\": rpc error: code = NotFound desc = could not find container \"32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f\": container with ID starting with 32c7cd510e3b633b7586d865a7ed9a28d7e472f3c799e9b4914961b9ce3c040f not found: ID does not exist"
Oct 30 22:22:10.228106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.228068    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:22:10.274573 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.274480    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-778d39d3/e2e-lifecycle-model-778d39d3-kserve-77644d9cdf-jzjb5"]
Oct 30 22:22:10.336679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:10.336632    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" podStartSLOduration=2.336619475 podStartE2EDuration="2.336619475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:10.334083584 +0000 UTC m=+2339.317990461" watchObservedRunningTime="2026-10-30 22:22:10.336619475 +0000 UTC m=+2339.320526350"
Oct 30 22:22:11.127024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:11.126989    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" path="/var/lib/kubelet/pods/41a4ee57-fafc-4e90-bfba-2e1811ae0f5e/volumes"
Oct 30 22:22:13.281597 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:13.281557    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:22:13.477285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.477249    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:22:13.477931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.477915    5510 cpu_manager.go:414] "containerMap: removing container" podUID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" containerName="main"
Oct 30 22:22:13.478011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.477932    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" containerName="main"
Oct 30 22:22:13.478050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.478032    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a4ee57-fafc-4e90-bfba-2e1811ae0f5e" containerName="main"
Oct 30 22:22:13.482869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.482848    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.533387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.533319    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-84a3cf-serving-cert\""
Oct 30 22:22:13.583850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.583805    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:22:13.595699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.595666    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2nt\" (UniqueName: \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.595881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.595827    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.595962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.595880    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.596020 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.595976    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.697002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.696962    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2nt\" (UniqueName: \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.697197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.697058    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.697197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.697096    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.697197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.697124    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.697373 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:13.697266    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-serving-cert: secret "maas-api-e2e-inf-b-84a3cf-serving-cert" not found
Oct 30 22:22:13.697373 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:13.697326    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls podName:02863ada-09c7-4996-9148-4844e22554f2 nodeName:}" failed. No retries permitted until 2026-10-30 22:22:14.197310324 +0000 UTC m=+2343.181217191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls") pod "maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" (UID: "02863ada-09c7-4996-9148-4844e22554f2") : secret "maas-api-e2e-inf-b-84a3cf-serving-cert" not found
Oct 30 22:22:13.697917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.697888    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.699580 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.699556    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:13.701487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.701453    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:13.766605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:13.766521    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2nt\" (UniqueName: \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:14.206402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:14.206361    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:14.209093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:14.209067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:14.399321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:14.399287    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:14.778035 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:14.777947    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:22:14.784341 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:22:14.784309    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02863ada_09c7_4996_9148_4844e22554f2.slice/crio-19f32f9745963f2d6460ed2da0ef98e9e6d52eef7b0dd809aedc8a1cc03e177d WatchSource:0}: Error finding container 19f32f9745963f2d6460ed2da0ef98e9e6d52eef7b0dd809aedc8a1cc03e177d: Status 404 returned error can't find the container with id 19f32f9745963f2d6460ed2da0ef98e9e6d52eef7b0dd809aedc8a1cc03e177d
Oct 30 22:22:15.166260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:15.166168    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:15.188459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:15.188416    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerStarted","Data":"19f32f9745963f2d6460ed2da0ef98e9e6d52eef7b0dd809aedc8a1cc03e177d"}
Oct 30 22:22:15.464695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:15.464663    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:22:16.196303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:16.196210    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerStarted","Data":"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"}
Oct 30 22:22:16.196303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:16.196281    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:16.304402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:16.304348    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" podStartSLOduration=2.4348222440000002 podStartE2EDuration="3.304334492s" totalImagesPullingTime="869.512248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:16.302991866 +0000 UTC m=+2345.286898743" watchObservedRunningTime="2026-10-30 22:22:16.304334492 +0000 UTC m=+2345.288241367"
Oct 30 22:22:18.698870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:18.698834    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:21.092147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:21.091942    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:22:22.203611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:22.203562    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.208:8443/health\": dial tcp 10.133.0.208:8443: connect: connection refused"
Oct 30 22:22:23.592517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:23.592432    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:26.270592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:26.270558    5510 generic.go:356] "Generic (PLEG): container finished" podID="02863ada-09c7-4996-9148-4844e22554f2" containerID="447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1" exitCode=1
Oct 30 22:22:26.271082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:26.270598    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerDied","Data":"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"}
Oct 30 22:22:26.271129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:26.271086    5510 scope.go:122] "RemoveContainer" containerID="447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"
Oct 30 22:22:27.460233 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:27.460195    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:28.285850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:28.285818    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerStarted","Data":"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"}
Oct 30 22:22:28.286028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:28.286016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:28.543019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:28.542931    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:30.305235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:30.305194    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:31.289800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:31.289271    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:31.496421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:31.496387    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:32.259729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:32.259689    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:32.956724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:32.956690    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:22:32.957161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:32.956895    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" podUID="70e0c9b4-9c4f-4481-b490-1576fede6f14" containerName="maas-api" containerID="cri-o://b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db" gracePeriod=30
Oct 30 22:22:33.129246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.129207    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:33.312674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.312650    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:22:33.329240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.329210    5510 generic.go:356] "Generic (PLEG): container finished" podID="70e0c9b4-9c4f-4481-b490-1576fede6f14" containerID="b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db" exitCode=0
Oct 30 22:22:33.329389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.329243    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"
Oct 30 22:22:33.329389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.329288    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" event={"ID":"70e0c9b4-9c4f-4481-b490-1576fede6f14","Type":"ContainerDied","Data":"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"}
Oct 30 22:22:33.329389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.329344    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f" event={"ID":"70e0c9b4-9c4f-4481-b490-1576fede6f14","Type":"ContainerDied","Data":"fca8567ef8ca2853100a9194c33e717d430d850a318b3a6a794b0e7b0d248030"}
Oct 30 22:22:33.329389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.329373    5510 scope.go:122] "RemoveContainer" containerID="b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"
Oct 30 22:22:33.338945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.338927    5510 scope.go:122] "RemoveContainer" containerID="b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"
Oct 30 22:22:33.339212 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:33.339193    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db\": container with ID starting with b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db not found: ID does not exist" containerID="b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"
Oct 30 22:22:33.339290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.339222    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db"} err="failed to get container status \"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db\": rpc error: code = NotFound desc = could not find container \"b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db\": container with ID starting with b94c10bfa55a6ec7579ce8598848d6ff8e6c3881f16ccecaa9bbc3032e4294db not found: ID does not exist"
Oct 30 22:22:33.458815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.456803    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\" (UniqueName: \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\") pod \"70e0c9b4-9c4f-4481-b490-1576fede6f14\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") "
Oct 30 22:22:33.458815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.456981    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\") pod \"70e0c9b4-9c4f-4481-b490-1576fede6f14\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") "
Oct 30 22:22:33.458815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.457029    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") pod \"70e0c9b4-9c4f-4481-b490-1576fede6f14\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") "
Oct 30 22:22:33.458815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.457259    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\") pod \"70e0c9b4-9c4f-4481-b490-1576fede6f14\" (UID: \"70e0c9b4-9c4f-4481-b490-1576fede6f14\") "
Oct 30 22:22:33.468069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.465769    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca" pod "70e0c9b4-9c4f-4481-b490-1576fede6f14" (UID: "70e0c9b4-9c4f-4481-b490-1576fede6f14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:33.485654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.485618    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls" pod "70e0c9b4-9c4f-4481-b490-1576fede6f14" (UID: "70e0c9b4-9c4f-4481-b490-1576fede6f14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:33.491981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.491935    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs" pod "70e0c9b4-9c4f-4481-b490-1576fede6f14" (UID: "70e0c9b4-9c4f-4481-b490-1576fede6f14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:33.494841 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.494764    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm" pod "70e0c9b4-9c4f-4481-b490-1576fede6f14" (UID: "70e0c9b4-9c4f-4481-b490-1576fede6f14"). InnerVolumeSpecName "kube-api-access-r6zzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:33.587857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.578843    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zzm\" (UniqueName: \"kubernetes.io/projected/70e0c9b4-9c4f-4481-b490-1576fede6f14-kube-api-access-r6zzm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:33.587857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.578900    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:33.587857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.578918    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e0c9b4-9c4f-4481-b490-1576fede6f14-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:33.587857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.578936    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e0c9b4-9c4f-4481-b490-1576fede6f14-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:33.746627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.746594    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:22:33.751647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:33.751617    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e66edb-78486d7d55-zgk6f"]
Oct 30 22:22:34.297476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:34.297445    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:22:34.674579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:34.674489    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:34.929102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:34.929019    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:35.088189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:35.088158    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:35.126488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:35.126454    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e0c9b4-9c4f-4481-b490-1576fede6f14" path="/var/lib/kubelet/pods/70e0c9b4-9c4f-4481-b490-1576fede6f14/volumes"
Oct 30 22:22:35.653736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:35.653700    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:36.361828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:36.361791    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 22:22:36.466984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:36.466945    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:38.102682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:38.102651    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:38.257296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:38.257263    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:39.967326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:39.967290    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:41.578772 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:41.578738    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:41.737492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:41.737447    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:41.918066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:41.917990    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:22:41.918232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:41.918209    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy" containerID="cri-o://f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62" gracePeriod=30
Oct 30 22:22:41.932981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:41.932952    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.455822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:42.455770    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:43.281512 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:43.281474    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:22:44.559988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:44.559952    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 22:22:44.658109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:44.658067    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 22:22:44.940552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:44.940470    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:45.105425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:45.105396    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:45.268064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:45.268029    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:47.205658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.205630    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:22:47.257069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257033    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257101    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257126    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257147    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257270 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257192    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257286    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257312    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257398    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257457    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\") pod \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\" (UID: \"7c2a7f65-1d47-4584-85b9-238d9a98bedb\") "
Oct 30 22:22:47.257768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257537    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:47.257768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257738    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:47.258150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.257940    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:47.258150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258086    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:47.258325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258216    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.258325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258235    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.258325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258250    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.258325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258263    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.258617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.258582    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:47.259664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.259625    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:47.260051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.260028    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "kube-api-access-zzp54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:47.260220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.260191    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:47.260322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.260226    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo" pod "7c2a7f65-1d47-4584-85b9-238d9a98bedb" (UID: "7c2a7f65-1d47-4584-85b9-238d9a98bedb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:47.359310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.359272    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.359310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.359311    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzp54\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-kube-api-access-zzp54\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.359525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.359327    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2a7f65-1d47-4584-85b9-238d9a98bedb-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.359525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.359343    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.359525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.359356    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2a7f65-1d47-4584-85b9-238d9a98bedb-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:47.431598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.431560    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 22:22:47.466640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.466608    5510 generic.go:356] "Generic (PLEG): container finished" podID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerID="f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62" exitCode=0
Oct 30 22:22:47.466852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.466662    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerDied","Data":"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"}
Oct 30 22:22:47.466852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.466670    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"
Oct 30 22:22:47.466852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.466706    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj" event={"ID":"7c2a7f65-1d47-4584-85b9-238d9a98bedb","Type":"ContainerDied","Data":"9cfaab8a2c47d24e9b7740da43a3043789ef98838ec02742aac8a09cdd5de7b2"}
Oct 30 22:22:47.466852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.466727    5510 scope.go:122] "RemoveContainer" containerID="f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"
Oct 30 22:22:47.479478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.479454    5510 scope.go:122] "RemoveContainer" containerID="461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"
Oct 30 22:22:47.490231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.490205    5510 scope.go:122] "RemoveContainer" containerID="f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"
Oct 30 22:22:47.490546 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:47.490525    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62\": container with ID starting with f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62 not found: ID does not exist" containerID="f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"
Oct 30 22:22:47.490646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.490558    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62"} err="failed to get container status \"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62\": rpc error: code = NotFound desc = could not find container \"f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62\": container with ID starting with f4ee5dfb6ea87f8d207b20736d89d0a4669de931cbe865b6abc7c8ad2cdbfe62 not found: ID does not exist"
Oct 30 22:22:47.490646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.490582    5510 scope.go:122] "RemoveContainer" containerID="461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"
Oct 30 22:22:47.490888 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:22:47.490869    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737\": container with ID starting with 461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737 not found: ID does not exist" containerID="461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"
Oct 30 22:22:47.490958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.490897    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737"} err="failed to get container status \"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737\": rpc error: code = NotFound desc = could not find container \"461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737\": container with ID starting with 461795bbeed94b24473beb24ca95b00fea6f256defb034a148c7ca9eb8351737 not found: ID does not exist"
Oct 30 22:22:47.550157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.550124    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:47.663591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.663553    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:22:47.673446 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:47.673412    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e66edb-openshift-default-867577894f-kjwwj"]
Oct 30 22:22:48.401044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.401010    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:22:48.401547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.401255    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy" containerID="cri-o://a5f8f0cca02651d685b9e2af28cf89f8f1af3bf54996b382359a74b1f59ec381" gracePeriod=30
Oct 30 22:22:48.415850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.415821    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:22:48.416045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.416011    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy" containerID="cri-o://ab5d390422da8f5c910119724819e4ff0aa872816910622d92052d95a6ad602d" gracePeriod=30
Oct 30 22:22:48.426169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.426145    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:22:48.426334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.426316    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy" containerID="cri-o://1a4536d2c366f759e8060b81bfe03b118805ab39bef015629400ca4ae5a2c1b1" gracePeriod=30
Oct 30 22:22:48.443934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:48.443909    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:49.127391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:49.127354    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" path="/var/lib/kubelet/pods/7c2a7f65-1d47-4584-85b9-238d9a98bedb/volumes"
Oct 30 22:22:49.561239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:49.561200    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing" containerID="cri-o://60e40e13934f671aad5b57225db7c8a53e2e9794cde59ad61c26e160f0b7b095" gracePeriod=25
Oct 30 22:22:49.658532 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:49.658494    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing" containerID="cri-o://248750efeba8640fba482b9302ab7ac4e812b1f64e5605466f798bf1e3905399" gracePeriod=25
Oct 30 22:22:50.370472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:50.370440    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:51.383729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.383686    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" containerID="cri-o://7dd2b5504cc636fe48faa8633a4684e7ea85710f19c7e38e4e0c2b41a9097cd4" gracePeriod=45
Oct 30 22:22:51.530228 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.530191    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 22:22:51.608183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.608139    5510 generic.go:356] "Generic (PLEG): container finished" podID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerID="7dd2b5504cc636fe48faa8633a4684e7ea85710f19c7e38e4e0c2b41a9097cd4" exitCode=0
Oct 30 22:22:51.608373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.608279    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" event={"ID":"c1c725fa-33cd-4ad1-97dd-4a89faa5c758","Type":"ContainerDied","Data":"7dd2b5504cc636fe48faa8633a4684e7ea85710f19c7e38e4e0c2b41a9097cd4"}
Oct 30 22:22:51.645768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.645712    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:22:51.737338 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737305    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.737489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737412    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.737489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737440    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.737567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737511    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\" (UniqueName: \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.737636 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737619    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.737704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737686    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.737771 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737721    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\") pod \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\" (UID: \"c1c725fa-33cd-4ad1-97dd-4a89faa5c758\") "
Oct 30 22:22:51.738006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.737954    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.738099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.738013    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.738236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.738222    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.738276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.738239    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.738276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.738249    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.739692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.739670    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "kube-api-access-jwgck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:51.739858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.739837    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:51.739858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.739845    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm" pod "c1c725fa-33cd-4ad1-97dd-4a89faa5c758" (UID: "c1c725fa-33cd-4ad1-97dd-4a89faa5c758"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:51.839991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.839953    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.839991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.839990    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:51.840198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:51.840005    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgck\" (UniqueName: \"kubernetes.io/projected/c1c725fa-33cd-4ad1-97dd-4a89faa5c758-kube-api-access-jwgck\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:52.620300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:52.620268    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"
Oct 30 22:22:52.620762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:52.620270    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s" event={"ID":"c1c725fa-33cd-4ad1-97dd-4a89faa5c758","Type":"ContainerDied","Data":"daf2f33a096cc68220982d6ef0bdeedac4fd5d3115c7a89d1a2ff69506cbe454"}
Oct 30 22:22:52.620762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:52.620393    5510 scope.go:122] "RemoveContainer" containerID="7dd2b5504cc636fe48faa8633a4684e7ea85710f19c7e38e4e0c2b41a9097cd4"
Oct 30 22:22:52.774164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:52.774130    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 22:22:52.777775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:52.777751    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a07620/premium-w1-a07620-kserve-6cdf8587d7-l2c2s"]
Oct 30 22:22:53.125496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.125463    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" path="/var/lib/kubelet/pods/c1c725fa-33cd-4ad1-97dd-4a89faa5c758/volumes"
Oct 30 22:22:53.634442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.634407    5510 generic.go:356] "Generic (PLEG): container finished" podID="e5aa62ce-a010-424b-8957-9d42889a0793" containerID="ab5d390422da8f5c910119724819e4ff0aa872816910622d92052d95a6ad602d" exitCode=0
Oct 30 22:22:53.634896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.634474    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" event={"ID":"e5aa62ce-a010-424b-8957-9d42889a0793","Type":"ContainerDied","Data":"ab5d390422da8f5c910119724819e4ff0aa872816910622d92052d95a6ad602d"}
Oct 30 22:22:53.636987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.636965    5510 generic.go:356] "Generic (PLEG): container finished" podID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerID="a5f8f0cca02651d685b9e2af28cf89f8f1af3bf54996b382359a74b1f59ec381" exitCode=0
Oct 30 22:22:53.637103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.637074    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" event={"ID":"5c989a62-4da2-4eb5-9743-a38629eb116b","Type":"ContainerDied","Data":"a5f8f0cca02651d685b9e2af28cf89f8f1af3bf54996b382359a74b1f59ec381"}
Oct 30 22:22:53.638659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.638636    5510 generic.go:356] "Generic (PLEG): container finished" podID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerID="1a4536d2c366f759e8060b81bfe03b118805ab39bef015629400ca4ae5a2c1b1" exitCode=0
Oct 30 22:22:53.638757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.638683    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" event={"ID":"e1f81844-3014-4484-bd88-a4f9e909db5a","Type":"ContainerDied","Data":"1a4536d2c366f759e8060b81bfe03b118805ab39bef015629400ca4ae5a2c1b1"}
Oct 30 22:22:53.719289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.719259    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:22:53.756453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.756420    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:22:53.764700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764673    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.764852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764809    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.764975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764886    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.764975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764936    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.764975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764957    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.765156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.764969    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.765156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765070    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.765156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765076    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.765156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765162    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.765444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765232    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.765444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765280    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.765444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765291    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\") pod \"e5aa62ce-a010-424b-8957-9d42889a0793\" (UID: \"e5aa62ce-a010-424b-8957-9d42889a0793\") "
Oct 30 22:22:53.765623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765523    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.765799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.765763    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.766022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.766002    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.766125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.766029    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.766125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.766048    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5aa62ce-a010-424b-8957-9d42889a0793-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.766125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.766062    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.766125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.766076    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.767378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.767357    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.767753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.767735    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "kube-api-access-kxc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.768677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.768648    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.768846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.768827    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token" pod "e5aa62ce-a010-424b-8957-9d42889a0793" (UID: "e5aa62ce-a010-424b-8957-9d42889a0793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.791113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.791094    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:22:53.867034 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867006    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.867187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867072    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867094    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.867187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867132    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867201    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.867425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867248    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867320    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867354    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867389    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.867697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867438    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867501    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867535    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.867697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867632    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\") pod \"5c989a62-4da2-4eb5-9743-a38629eb116b\" (UID: \"5c989a62-4da2-4eb5-9743-a38629eb116b\") "
Oct 30 22:22:53.867697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867667    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.868043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867723    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.868043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867727    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.868043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867757    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.868043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.867864    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\") pod \"e1f81844-3014-4484-bd88-a4f9e909db5a\" (UID: \"e1f81844-3014-4484-bd88-a4f9e909db5a\") "
Oct 30 22:22:53.868261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868134    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868324    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868549    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868556    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868567    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868589    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5aa62ce-a010-424b-8957-9d42889a0793-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868605    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxc8l\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-kube-api-access-kxc8l\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868622    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5aa62ce-a010-424b-8957-9d42889a0793-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868636    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c989a62-4da2-4eb5-9743-a38629eb116b-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868665    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5aa62ce-a010-424b-8957-9d42889a0793-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868678    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868755    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.868928    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.869153    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.869203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.869173    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:53.870093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.869217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.870151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.870093    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.870490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.870450    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.870570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.870540    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "kube-api-access-974fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.871131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.871105    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.871238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.871150    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.871554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.871530    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy" pod "e1f81844-3014-4484-bd88-a4f9e909db5a" (UID: "e1f81844-3014-4484-bd88-a4f9e909db5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:53.871948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.871930    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "kube-api-access-wcsmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:53.872552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.872537    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo" pod "5c989a62-4da2-4eb5-9743-a38629eb116b" (UID: "5c989a62-4da2-4eb5-9743-a38629eb116b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:53.970052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970013    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970051    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970066    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970079    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970093    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970106    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970121    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970137    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970150    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f81844-3014-4484-bd88-a4f9e909db5a-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970164    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsmf\" (UniqueName: \"kubernetes.io/projected/5c989a62-4da2-4eb5-9743-a38629eb116b-kube-api-access-wcsmf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970178    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974fx\" (UniqueName: \"kubernetes.io/projected/e1f81844-3014-4484-bd88-a4f9e909db5a-kube-api-access-974fx\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970193    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970207    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970221    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c989a62-4da2-4eb5-9743-a38629eb116b-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:53.970309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:53.970235    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f81844-3014-4484-bd88-a4f9e909db5a-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:54.646908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.646873    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"
Oct 30 22:22:54.646908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.646894    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm" event={"ID":"e1f81844-3014-4484-bd88-a4f9e909db5a","Type":"ContainerDied","Data":"cf8a2e6a02e762fecd2398bc7e0c74dc02131fb071b8e1852e0cc602c25270a4"}
Oct 30 22:22:54.647409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.646963    5510 scope.go:122] "RemoveContainer" containerID="1a4536d2c366f759e8060b81bfe03b118805ab39bef015629400ca4ae5a2c1b1"
Oct 30 22:22:54.648368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.648349    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"
Oct 30 22:22:54.648475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.648393    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t" event={"ID":"e5aa62ce-a010-424b-8957-9d42889a0793","Type":"ContainerDied","Data":"9971cf7a5c81a1433213a5d788eb6188bac4285a3172f1f7a955e4c595d94db7"}
Oct 30 22:22:54.650044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.650018    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn" event={"ID":"5c989a62-4da2-4eb5-9743-a38629eb116b","Type":"ContainerDied","Data":"b41677b4ed39c030d5d8566644bca6aba463a8f6be42c3742fba6d037de6858d"}
Oct 30 22:22:54.650149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.650033    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"
Oct 30 22:22:54.663897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.663816    5510 scope.go:122] "RemoveContainer" containerID="ab5d390422da8f5c910119724819e4ff0aa872816910622d92052d95a6ad602d"
Oct 30 22:22:54.673591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.673568    5510 scope.go:122] "RemoveContainer" containerID="a5f8f0cca02651d685b9e2af28cf89f8f1af3bf54996b382359a74b1f59ec381"
Oct 30 22:22:54.801552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.801512    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:22:54.847536 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.847502    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-ncpkn"]
Oct 30 22:22:54.917632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.917595    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:22:54.922757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.922728    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-sntpm"]
Oct 30 22:22:54.991700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:54.991658    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 22:22:55.024328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.024291    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:22:55.026694 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.026670    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778d39d3-openshift-default-bff74755c-5sw4t"]
Oct 30 22:22:55.125811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.125758    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" path="/var/lib/kubelet/pods/5c989a62-4da2-4eb5-9743-a38629eb116b/volumes"
Oct 30 22:22:55.126437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.126420    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" path="/var/lib/kubelet/pods/e1f81844-3014-4484-bd88-a4f9e909db5a/volumes"
Oct 30 22:22:55.127729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.127691    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" path="/var/lib/kubelet/pods/e5aa62ce-a010-424b-8957-9d42889a0793/volumes"
Oct 30 22:22:55.349924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.349836    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:55.528888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:55.528845    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.681271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:57.681230    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:22:57.681842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:57.681240    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:22:59.992527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:22:59.992482    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing" containerID="cri-o://fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345" gracePeriod=25
Oct 30 22:23:00.333017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:00.332941    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.454007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.453934    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main" containerID="cri-o://4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91" gracePeriod=45
Oct 30 22:23:02.735952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.735920    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:23:02.744252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.744219    5510 generic.go:356] "Generic (PLEG): container finished" podID="baf17584-09fb-41a8-a269-e165ed0048ed" containerID="4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91" exitCode=0
Oct 30 22:23:02.744424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.744260    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"
Oct 30 22:23:02.744424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.744301    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" event={"ID":"baf17584-09fb-41a8-a269-e165ed0048ed","Type":"ContainerDied","Data":"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"}
Oct 30 22:23:02.744424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.744345    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg" event={"ID":"baf17584-09fb-41a8-a269-e165ed0048ed","Type":"ContainerDied","Data":"6e38134085b0ccd95e1a3ac99b8a4a3a97c076b8d5b750aa03cec7a28a83d6d1"}
Oct 30 22:23:02.744424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.744365    5510 scope.go:122] "RemoveContainer" containerID="4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"
Oct 30 22:23:02.756108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.756089    5510 scope.go:122] "RemoveContainer" containerID="4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"
Oct 30 22:23:02.756442 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:02.756420    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91\": container with ID starting with 4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91 not found: ID does not exist" containerID="4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"
Oct 30 22:23:02.756518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.756451    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91"} err="failed to get container status \"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91\": rpc error: code = NotFound desc = could not find container \"4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91\": container with ID starting with 4f05b410bbe5ac10d667ad2c7d3c0b6dc20452380eb9c14c3b0d6e021ee36f91 not found: ID does not exist"
Oct 30 22:23:02.798018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.797984    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798041    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798080    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798119    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798204    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.798458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798213    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798351    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\" (UniqueName: \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\") pod \"baf17584-09fb-41a8-a269-e165ed0048ed\" (UID: \"baf17584-09fb-41a8-a269-e165ed0048ed\") "
Oct 30 22:23:02.798458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798350    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.798458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798393    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.798801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798763    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.798934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798835    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.798934 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.798851    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.800375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.800347    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:02.800484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.800418    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:02.800484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.800421    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn" pod "baf17584-09fb-41a8-a269-e165ed0048ed" (UID: "baf17584-09fb-41a8-a269-e165ed0048ed"). InnerVolumeSpecName "kube-api-access-jjgdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:02.901422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.901389    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf17584-09fb-41a8-a269-e165ed0048ed-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.901422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.901420    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf17584-09fb-41a8-a269-e165ed0048ed-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:02.901698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:02.901435    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgdn\" (UniqueName: \"kubernetes.io/projected/baf17584-09fb-41a8-a269-e165ed0048ed-kube-api-access-jjgdn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:03.159432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:03.159393    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 22:23:03.163433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:03.163399    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5be0d1/simulator-w2-5be0d1-kserve-6867554d6b-xxkzg"]
Oct 30 22:23:04.442298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:04.442262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.126432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.126394    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" path="/var/lib/kubelet/pods/baf17584-09fb-41a8-a269-e165ed0048ed/volumes"
Oct 30 22:23:05.998030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.997991    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:23:05.998572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998559    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:23:05.998572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998572    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998583    5510 cpu_manager.go:414] "containerMap: removing container" podUID="70e0c9b4-9c4f-4481-b490-1576fede6f14" containerName="maas-api"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998588    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e0c9b4-9c4f-4481-b490-1576fede6f14" containerName="maas-api"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998609    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998615    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998624    5510 cpu_manager.go:414] "containerMap: removing container" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main"
Oct 30 22:23:05.998656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998629    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998659    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998664    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998689    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998693    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998700    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998704    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998713    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998717    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998804    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998814    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e0c9b4-9c4f-4481-b490-1576fede6f14" containerName="maas-api"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998823    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c989a62-4da2-4eb5-9743-a38629eb116b" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998830    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf17584-09fb-41a8-a269-e165ed0048ed" containerName="main"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998838    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aa62ce-a010-424b-8957-9d42889a0793" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998847    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f81844-3014-4484-bd88-a4f9e909db5a" containerName="istio-proxy"
Oct 30 22:23:05.998880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:05.998853    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c725fa-33cd-4ad1-97dd-4a89faa5c758" containerName="main"
Oct 30 22:23:06.008362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.008335    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.062299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.062261    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:23:06.104586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.104553    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c1e2ab-serving-cert\""
Oct 30 22:23:06.109161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.109130    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.169682 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.169650    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.169845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.169810    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.169930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.169902    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrcl\" (UniqueName: \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.170073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.170048    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.281830    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.282151    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrcl\" (UniqueName: \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.282225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.282301    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.286147    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.288929 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:06.288925    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-serving-cert: secret "maas-api-e2e-api-b-c1e2ab-serving-cert" not found
Oct 30 22:23:06.289426 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:06.289010    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls podName:1f980b9b-c2b1-47be-aad4-f1b0ce244d13 nodeName:}" failed. No retries permitted until 2026-10-30 22:23:06.788991725 +0000 UTC m=+2395.772898592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls") pod "maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" (UID: "1f980b9b-c2b1-47be-aad4-f1b0ce244d13") : secret "maas-api-e2e-api-b-c1e2ab-serving-cert" not found
Oct 30 22:23:06.289727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.289607    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.292479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.292452    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.353095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.353065    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrcl\" (UniqueName: \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.487314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.487283    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.771558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.771525    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.889189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.889152    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.891659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.891626    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") pod \"maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:06.923322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:06.923279    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:07.302800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:07.302749    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:23:07.309930 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:23:07.309898    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f980b9b_c2b1_47be_aad4_f1b0ce244d13.slice/crio-cc97cc639a87db49858a3099a4ed206236d4e4c6bacd74850d34765893387980 WatchSource:0}: Error finding container cc97cc639a87db49858a3099a4ed206236d4e4c6bacd74850d34765893387980: Status 404 returned error can't find the container with id cc97cc639a87db49858a3099a4ed206236d4e4c6bacd74850d34765893387980
Oct 30 22:23:07.681815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:07.681715    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:07.681815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:07.681748    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:07.797148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:07.797111    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" event={"ID":"1f980b9b-c2b1-47be-aad4-f1b0ce244d13","Type":"ContainerStarted","Data":"cc97cc639a87db49858a3099a4ed206236d4e4c6bacd74850d34765893387980"}
Oct 30 22:23:08.806658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:08.806624    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" event={"ID":"1f980b9b-c2b1-47be-aad4-f1b0ce244d13","Type":"ContainerStarted","Data":"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"}
Oct 30 22:23:08.807103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:08.806750    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:08.917832 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:08.917767    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" podStartSLOduration=3.027347866 podStartE2EDuration="3.917752492s" totalImagesPullingTime="890.404626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:08.915534245 +0000 UTC m=+2397.899441122" watchObservedRunningTime="2026-10-30 22:23:08.917752492 +0000 UTC m=+2397.901659368"
Oct 30 22:23:09.802223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.802187    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:09.828766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.828691    5510 generic.go:356] "Generic (PLEG): container finished" podID="127344d8-b678-4c8e-935d-36d812f108e5" containerID="60e40e13934f671aad5b57225db7c8a53e2e9794cde59ad61c26e160f0b7b095" exitCode=0
Oct 30 22:23:09.829291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.828757    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" event={"ID":"127344d8-b678-4c8e-935d-36d812f108e5","Type":"ContainerDied","Data":"60e40e13934f671aad5b57225db7c8a53e2e9794cde59ad61c26e160f0b7b095"}
Oct 30 22:23:09.829291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.828860    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h" event={"ID":"127344d8-b678-4c8e-935d-36d812f108e5","Type":"ContainerDied","Data":"27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933"}
Oct 30 22:23:09.829291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.828882    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27407bc7a43b3243fa59ef62e55edaa5aaa82124d7ac877ff735312eee6ca933"
Oct 30 22:23:09.830816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.830792    5510 generic.go:356] "Generic (PLEG): container finished" podID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerID="248750efeba8640fba482b9302ab7ac4e812b1f64e5605466f798bf1e3905399" exitCode=0
Oct 30 22:23:09.830922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.830850    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" event={"ID":"bfdd14ec-7908-4321-848c-c51cf2f14e7f","Type":"ContainerDied","Data":"248750efeba8640fba482b9302ab7ac4e812b1f64e5605466f798bf1e3905399"}
Oct 30 22:23:09.831965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.831945    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 22:23:09.925182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.925156    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 22:23:09.932268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932241    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\" (UniqueName: \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\") pod \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") "
Oct 30 22:23:09.932409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932300    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\" (UniqueName: \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\") pod \"127344d8-b678-4c8e-935d-36d812f108e5\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") "
Oct 30 22:23:09.932409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932398    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\") pod \"127344d8-b678-4c8e-935d-36d812f108e5\" (UID: \"127344d8-b678-4c8e-935d-36d812f108e5\") "
Oct 30 22:23:09.932527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932440    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\") pod \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\" (UID: \"bfdd14ec-7908-4321-848c-c51cf2f14e7f\") "
Oct 30 22:23:09.932837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932809    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume" pod "127344d8-b678-4c8e-935d-36d812f108e5" (UID: "127344d8-b678-4c8e-935d-36d812f108e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:09.933000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.932862    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume" pod "bfdd14ec-7908-4321-848c-c51cf2f14e7f" (UID: "bfdd14ec-7908-4321-848c-c51cf2f14e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:09.933108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.933085    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127344d8-b678-4c8e-935d-36d812f108e5-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.933217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.933112    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdd14ec-7908-4321-848c-c51cf2f14e7f-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:09.934406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.934384    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz" pod "127344d8-b678-4c8e-935d-36d812f108e5" (UID: "127344d8-b678-4c8e-935d-36d812f108e5"). InnerVolumeSpecName "kube-api-access-mz9sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:09.934480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:09.934465    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr" pod "bfdd14ec-7908-4321-848c-c51cf2f14e7f" (UID: "bfdd14ec-7908-4321-848c-c51cf2f14e7f"). InnerVolumeSpecName "kube-api-access-m44lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:10.034497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.034457    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44lr\" (UniqueName: \"kubernetes.io/projected/bfdd14ec-7908-4321-848c-c51cf2f14e7f-kube-api-access-m44lr\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.034497 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.034495    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz9sz\" (UniqueName: \"kubernetes.io/projected/127344d8-b678-4c8e-935d-36d812f108e5-kube-api-access-mz9sz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:10.839048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.838969    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"
Oct 30 22:23:10.839048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.839003    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf" event={"ID":"bfdd14ec-7908-4321-848c-c51cf2f14e7f","Type":"ContainerDied","Data":"b1e2e76554e3a0f5df5beb79d2339ce38caa9edd26141d500cb9c547a6d919ec"}
Oct 30 22:23:10.839515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.839065    5510 scope.go:122] "RemoveContainer" containerID="248750efeba8640fba482b9302ab7ac4e812b1f64e5605466f798bf1e3905399"
Oct 30 22:23:10.839515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.839247    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"
Oct 30 22:23:10.993420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:10.993389    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 22:23:11.039319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.039286    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a07620-75c9887dcf-tq4gf"]
Oct 30 22:23:11.100476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.100391    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 22:23:11.106383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.106356    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a07620-f94d84c5b-cqf6h"]
Oct 30 22:23:11.125108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.125079    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127344d8-b678-4c8e-935d-36d812f108e5" path="/var/lib/kubelet/pods/127344d8-b678-4c8e-935d-36d812f108e5/volumes"
Oct 30 22:23:11.125541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.125526    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" path="/var/lib/kubelet/pods/bfdd14ec-7908-4321-848c-c51cf2f14e7f/volumes"
Oct 30 22:23:11.770088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.770056    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.777264 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:11.777241    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:23:12.292453 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:12.292424    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:23:13.281934 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:13.281900    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:23:14.836467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:14.836439    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:23:16.549380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:16.549324    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.008897 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:17.008864    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.666751 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:19.666717    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:19.802188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:19.802147    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:23:19.834919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:19.834884    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.270616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.270552    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 22:23:20.371890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.371847    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\" (UniqueName: \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\") pod \"67065ea8-6706-4f52-991f-b7153cf55f6e\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") "
Oct 30 22:23:20.372059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.371940    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\") pod \"67065ea8-6706-4f52-991f-b7153cf55f6e\" (UID: \"67065ea8-6706-4f52-991f-b7153cf55f6e\") "
Oct 30 22:23:20.372362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.372317    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume" pod "67065ea8-6706-4f52-991f-b7153cf55f6e" (UID: "67065ea8-6706-4f52-991f-b7153cf55f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:20.372509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.372455    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67065ea8-6706-4f52-991f-b7153cf55f6e-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:20.374041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.374016    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb" pod "67065ea8-6706-4f52-991f-b7153cf55f6e" (UID: "67065ea8-6706-4f52-991f-b7153cf55f6e"). InnerVolumeSpecName "kube-api-access-58xdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:20.474357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.474316    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xdb\" (UniqueName: \"kubernetes.io/projected/67065ea8-6706-4f52-991f-b7153cf55f6e-kube-api-access-58xdb\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:20.549452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.549378    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.923637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.923544    5510 generic.go:356] "Generic (PLEG): container finished" podID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerID="fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345" exitCode=0
Oct 30 22:23:20.923637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.923589    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"
Oct 30 22:23:20.923637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.923624    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" event={"ID":"67065ea8-6706-4f52-991f-b7153cf55f6e","Type":"ContainerDied","Data":"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"}
Oct 30 22:23:20.924218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.923674    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl" event={"ID":"67065ea8-6706-4f52-991f-b7153cf55f6e","Type":"ContainerDied","Data":"9e296a6a961dc28f7db6cd607cee8c92020db9c592681048a35d062b9289bbb1"}
Oct 30 22:23:20.924218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.923696    5510 scope.go:122] "RemoveContainer" containerID="fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"
Oct 30 22:23:20.933908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.933890    5510 scope.go:122] "RemoveContainer" containerID="fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"
Oct 30 22:23:20.934171 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:20.934149    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345\": container with ID starting with fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345 not found: ID does not exist" containerID="fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"
Oct 30 22:23:20.934272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:20.934176    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345"} err="failed to get container status \"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345\": rpc error: code = NotFound desc = could not find container \"fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345\": container with ID starting with fbdf70bb86481aae0cb7487c9cc9d5c219a719abcd3fecdc3ce35f35a6403345 not found: ID does not exist"
Oct 30 22:23:21.077201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:21.077160    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 22:23:21.135200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:21.135169    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5be0d1-5845f89868-8c8zl"]
Oct 30 22:23:21.644966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:21.644929    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:22.459018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:22.458978    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:23:23.125495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:23.125462    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" path="/var/lib/kubelet/pods/67065ea8-6706-4f52-991f-b7153cf55f6e/volumes"
Oct 30 22:23:23.355184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:23.355154    5510 scope.go:122] "RemoveContainer" containerID="60e40e13934f671aad5b57225db7c8a53e2e9794cde59ad61c26e160f0b7b095"
Oct 30 22:23:24.322819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:24.322110    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.516416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:24.514895    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:24.810813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:24.774305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.230801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:25.228344    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.955451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:25.955404    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:25.956107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:25.956068    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.097805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.096385    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 22:23:26.171812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.141144    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.389989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.389918    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.444523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.444469    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:23:26.448901 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.448863    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.509938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.509896    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:26.510278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.510245    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:26.599318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.599282    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.616802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.613899    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4c796 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:26.616802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:26.614024    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.150160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.150115    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.151333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.151307    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:23:27.228275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.228180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.420329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.420199    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded"
Oct 30 22:23:27.423471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.423439    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:23:27.429081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.421470    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.468803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.455871    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:23:27.468803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.456301    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:23:27.468803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.463864    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:27.468803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.464888    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:23:27.517289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.517164    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.659803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.648834    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:23:27.698862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.668956    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:23:27.731604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.731400    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:23:27.731604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:27.731466    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:23:28.951105 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:28.951047    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:29.120659 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:29.120626    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.378218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:30.378177    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:30.708900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:30.708869    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:32.258400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:32.258360    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:33.596714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:33.596670    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:34.065940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:34.065905    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:34.066134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:34.065978    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.231746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.231717    5510 generic.go:356] "Generic (PLEG): container finished" podID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerID="ef988c80068dd083c8ff21f475d59b2a0e9d6df4f296df636a4ac51d5cd0f45f" exitCode=0
Oct 30 22:23:35.232242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.231769    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerDied","Data":"ef988c80068dd083c8ff21f475d59b2a0e9d6df4f296df636a4ac51d5cd0f45f"}
Oct 30 22:23:35.232242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.232148    5510 scope.go:122] "RemoveContainer" containerID="ef988c80068dd083c8ff21f475d59b2a0e9d6df4f296df636a4ac51d5cd0f45f"
Oct 30 22:23:35.233687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.233651    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:35.233830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.233739    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:35.233830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:35.233808    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:23:36.239774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:36.239735    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerStarted","Data":"7be10dd101a6d5c5144c70df423254cb5d52e6210b7a49224d9962eb9ea37fe4"}
Oct 30 22:23:36.995968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:36.995924    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:23:36.996187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:36.996131    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:36.996187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:36.996177    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:37.198243 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:37.198206    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:37.996278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:37.996241    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:37.996992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:37.996952    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:38.018107 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:38.018079    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 22:23:38.283685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:38.283594    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:38.436612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:38.436574    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:38.995979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:38.995939    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:38.996193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:38.996014    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:39.996104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:39.996066    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:39.996621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:39.996123    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:40.996210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:40.996171    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:40.996631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:40.996245    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:41.996293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:41.996250    5510 patch_prober.go:29] interesting pod/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:23:41.996719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:41.996317    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:23:42.070723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:42.070685    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:42.999769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:42.999733    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:23:43.000252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.000178    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:23:43.000996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.000977    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:23:43.019092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.019063    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" podUID="b902d615-4402-4008-95ee-b048ca9be731" containerName="payload-pre-processing" containerID="cri-o://4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a" gracePeriod=25
Oct 30 22:23:43.285826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.285755    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 22:23:43.300412 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:43.300344    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:23:43.317365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.317337    5510 generic.go:356] "Generic (PLEG): container finished" podID="b902d615-4402-4008-95ee-b048ca9be731" containerID="4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a" exitCode=0
Oct 30 22:23:43.317683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.317651    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"
Oct 30 22:23:43.317853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.317748    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" event={"ID":"b902d615-4402-4008-95ee-b048ca9be731","Type":"ContainerDied","Data":"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"}
Oct 30 22:23:43.317853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.317806    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx" event={"ID":"b902d615-4402-4008-95ee-b048ca9be731","Type":"ContainerDied","Data":"f4f96cfa736658666f8c98b72da2468b910c83bae27767fa7fcb15eb35843ed2"}
Oct 30 22:23:43.317853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.317835    5510 scope.go:122] "RemoveContainer" containerID="4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"
Oct 30 22:23:43.332648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.332527    5510 scope.go:122] "RemoveContainer" containerID="4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"
Oct 30 22:23:43.332955 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:23:43.332930    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a\": container with ID starting with 4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a not found: ID does not exist" containerID="4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"
Oct 30 22:23:43.333068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.332967    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a"} err="failed to get container status \"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a\": rpc error: code = NotFound desc = could not find container \"4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a\": container with ID starting with 4d11daa4590aad755407c2fb2681ecdf0d3cf7be9d4b2e168111a4452554235a not found: ID does not exist"
Oct 30 22:23:43.443231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.443200    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\") pod \"b902d615-4402-4008-95ee-b048ca9be731\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") "
Oct 30 22:23:43.443475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.443453    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\" (UniqueName: \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\") pod \"b902d615-4402-4008-95ee-b048ca9be731\" (UID: \"b902d615-4402-4008-95ee-b048ca9be731\") "
Oct 30 22:23:43.443621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.443601    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume" pod "b902d615-4402-4008-95ee-b048ca9be731" (UID: "b902d615-4402-4008-95ee-b048ca9be731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:43.444038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.444009    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902d615-4402-4008-95ee-b048ca9be731-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:43.445557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.445530    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl" pod "b902d615-4402-4008-95ee-b048ca9be731" (UID: "b902d615-4402-4008-95ee-b048ca9be731"). InnerVolumeSpecName "kube-api-access-6xghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:43.545382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.545305    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xghl\" (UniqueName: \"kubernetes.io/projected/b902d615-4402-4008-95ee-b048ca9be731-kube-api-access-6xghl\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:43.732444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.732411    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 22:23:43.738125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:43.738102    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2d1f91-77dfb86795-gnrbx"]
Oct 30 22:23:45.125285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:45.125252    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b902d615-4402-4008-95ee-b048ca9be731" path="/var/lib/kubelet/pods/b902d615-4402-4008-95ee-b048ca9be731/volumes"
Oct 30 22:23:46.919935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:46.919892    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:52.050186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:52.050086    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:56.874742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:23:56.874706    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:01.659073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:01.659026    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.746478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:05.746441    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:24:06.501942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:06.501914    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.766756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:07.766716    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:07.934935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:07.934906    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:09.677825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:09.677776    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:09.832115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:09.832084    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:11.331594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:11.331565    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.300473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:12.300434    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:12.957382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:12.957344    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:13.269465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:13.269432    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:13.280663 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:13.280636    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:24:13.432658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:13.432622    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:13.990382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:13.990341    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:14.881749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:14.881715    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:15.986479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:15.986438    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:16.187961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:16.187923    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:16.354978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:16.354898    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:16.517599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:16.517564    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:17.243159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:17.243124    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:18.460374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:18.460338    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.214132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:20.211083    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.396110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:20.396079    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:20.525232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:20.525152    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 22:24:20.525409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:20.525352    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy" containerID="cri-o://0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c" gracePeriod=30
Oct 30 22:24:20.637655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:20.637619    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.696397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:23.696315    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.847998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:23.847969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:25.783298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.783268    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 22:24:25.900222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900142    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900241    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900293    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900343    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900421    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:25.900572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900439    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900477    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900516    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900570    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900581    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:25.900900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900657    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\") pod \"3879e9b5-fece-49a0-a747-e56664ef2df4\" (UID: \"3879e9b5-fece-49a0-a747-e56664ef2df4\") "
Oct 30 22:24:25.900900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.900679    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:25.901102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.901069    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:25.901187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.901134    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:25.901187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.901154    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:25.901187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.901169    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879e9b5-fece-49a0-a747-e56664ef2df4-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:25.901371 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.901348    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:25.903238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.903217    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:25.903238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.903227    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:25.903369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.903283    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "kube-api-access-6dztg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:25.903369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:25.903319    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token" pod "3879e9b5-fece-49a0-a747-e56664ef2df4" (UID: "3879e9b5-fece-49a0-a747-e56664ef2df4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:26.002313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002281    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.002313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002311    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.002560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002323    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.002560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002333    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dztg\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-kube-api-access-6dztg\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.002560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002346    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879e9b5-fece-49a0-a747-e56664ef2df4-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.002560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.002359    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879e9b5-fece-49a0-a747-e56664ef2df4-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:26.591608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.591566    5510 generic.go:356] "Generic (PLEG): container finished" podID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerID="0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c" exitCode=0
Oct 30 22:24:26.591914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.591618    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"
Oct 30 22:24:26.591914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.591644    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerDied","Data":"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"}
Oct 30 22:24:26.591914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.591708    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg" event={"ID":"3879e9b5-fece-49a0-a747-e56664ef2df4","Type":"ContainerDied","Data":"e55521df84080c0e3a1cd11c0ebefa8a0182becc48250337949b077f93d21c7a"}
Oct 30 22:24:26.591914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.591739    5510 scope.go:122] "RemoveContainer" containerID="0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"
Oct 30 22:24:26.603128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.603111    5510 scope.go:122] "RemoveContainer" containerID="0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"
Oct 30 22:24:26.612442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.612425    5510 scope.go:122] "RemoveContainer" containerID="0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"
Oct 30 22:24:26.612681 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:26.612664    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c\": container with ID starting with 0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c not found: ID does not exist" containerID="0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"
Oct 30 22:24:26.612734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.612691    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c"} err="failed to get container status \"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c\": rpc error: code = NotFound desc = could not find container \"0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c\": container with ID starting with 0285876e388a10c89765d0ca289107adca48cf607ed24c371cee26fe68f45f7c not found: ID does not exist"
Oct 30 22:24:26.612734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.612708    5510 scope.go:122] "RemoveContainer" containerID="0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"
Oct 30 22:24:26.612927 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:26.612915    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0\": container with ID starting with 0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0 not found: ID does not exist" containerID="0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"
Oct 30 22:24:26.612969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.612935    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0"} err="failed to get container status \"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0\": rpc error: code = NotFound desc = could not find container \"0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0\": container with ID starting with 0a791f39355b6306f95f2d87fdb3d80ae7887e0ed44c4dec9d64feb519c38fc0 not found: ID does not exist"
Oct 30 22:24:26.745089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.745056    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 22:24:26.792465 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:26.792431    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a07620-openshift-default-8d8c5c5b7-b98gg"]
Oct 30 22:24:27.126959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:27.126920    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" path="/var/lib/kubelet/pods/3879e9b5-fece-49a0-a747-e56664ef2df4/volumes"
Oct 30 22:24:28.867341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:28.867305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:29.831100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:29.831057    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:29.888440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:29.886642    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:24:29.994474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:29.994372    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:24:30.085621 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:30.085541    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.252845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:30.252806    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:30.414504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:30.414423    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.047045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:31.047014    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:33.876517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:33.876442    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:34.889602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:34.889528    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" podUID="87429923-6520-4079-9c03-88d308793ffa" containerName="payload-processing" containerID="cri-o://38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665" gracePeriod=25
Oct 30 22:24:34.995618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:34.995581    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" podUID="6b326b3f-8697-46f6-ace0-e167448c9a0d" containerName="payload-pre-processing" containerID="cri-o://e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915" gracePeriod=25
Oct 30 22:24:35.145882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.145810    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:24:35.174511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.174480    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\") pod \"87429923-6520-4079-9c03-88d308793ffa\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") "
Oct 30 22:24:35.174671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.174548    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\" (UniqueName: \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\") pod \"87429923-6520-4079-9c03-88d308793ffa\" (UID: \"87429923-6520-4079-9c03-88d308793ffa\") "
Oct 30 22:24:35.175432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.175384    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume" pod "87429923-6520-4079-9c03-88d308793ffa" (UID: "87429923-6520-4079-9c03-88d308793ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.178472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.178447    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz" pod "87429923-6520-4079-9c03-88d308793ffa" (UID: "87429923-6520-4079-9c03-88d308793ffa"). InnerVolumeSpecName "kube-api-access-qf7zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.230763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.230739    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:24:35.275667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.275639    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\" (UniqueName: \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\") pod \"6b326b3f-8697-46f6-ace0-e167448c9a0d\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") "
Oct 30 22:24:35.275853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.275751    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\") pod \"6b326b3f-8697-46f6-ace0-e167448c9a0d\" (UID: \"6b326b3f-8697-46f6-ace0-e167448c9a0d\") "
Oct 30 22:24:35.276126 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.276102    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume" pod "6b326b3f-8697-46f6-ace0-e167448c9a0d" (UID: "6b326b3f-8697-46f6-ace0-e167448c9a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.276201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.276141    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87429923-6520-4079-9c03-88d308793ffa-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.276201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.276164    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7zz\" (UniqueName: \"kubernetes.io/projected/87429923-6520-4079-9c03-88d308793ffa-kube-api-access-qf7zz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.277668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.277647    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc" pod "6b326b3f-8697-46f6-ace0-e167448c9a0d" (UID: "6b326b3f-8697-46f6-ace0-e167448c9a0d"). InnerVolumeSpecName "kube-api-access-jdhmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.376926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.376892    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhmc\" (UniqueName: \"kubernetes.io/projected/6b326b3f-8697-46f6-ace0-e167448c9a0d-kube-api-access-jdhmc\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.376926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.376921    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b326b3f-8697-46f6-ace0-e167448c9a0d-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.679545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.679511    5510 generic.go:356] "Generic (PLEG): container finished" podID="87429923-6520-4079-9c03-88d308793ffa" containerID="38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665" exitCode=0
Oct 30 22:24:35.679723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.679568    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"
Oct 30 22:24:35.679723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.679598    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" event={"ID":"87429923-6520-4079-9c03-88d308793ffa","Type":"ContainerDied","Data":"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"}
Oct 30 22:24:35.679723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.679650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs" event={"ID":"87429923-6520-4079-9c03-88d308793ffa","Type":"ContainerDied","Data":"988510bfc0c882d9ea2783ad87c4ead02454bec340a9b6b7dca5c395c77a3482"}
Oct 30 22:24:35.679723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.679678    5510 scope.go:122] "RemoveContainer" containerID="38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"
Oct 30 22:24:35.681349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.681259    5510 generic.go:356] "Generic (PLEG): container finished" podID="6b326b3f-8697-46f6-ace0-e167448c9a0d" containerID="e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915" exitCode=0
Oct 30 22:24:35.681349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.681287    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" event={"ID":"6b326b3f-8697-46f6-ace0-e167448c9a0d","Type":"ContainerDied","Data":"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"}
Oct 30 22:24:35.681349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.681325    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"
Oct 30 22:24:35.681349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.681342    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj" event={"ID":"6b326b3f-8697-46f6-ace0-e167448c9a0d","Type":"ContainerDied","Data":"a17ce93d72388312831faca7fe4666dacf2872e7268b4f8c43478f485cc88029"}
Oct 30 22:24:35.691501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.691411    5510 scope.go:122] "RemoveContainer" containerID="38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"
Oct 30 22:24:35.691740 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:35.691717    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665\": container with ID starting with 38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665 not found: ID does not exist" containerID="38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"
Oct 30 22:24:35.691853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.691749    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665"} err="failed to get container status \"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665\": rpc error: code = NotFound desc = could not find container \"38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665\": container with ID starting with 38069e6d6c19c4acaa969f5925832a523728c7bb5fe219be830fcc99c2469665 not found: ID does not exist"
Oct 30 22:24:35.691853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.691770    5510 scope.go:122] "RemoveContainer" containerID="e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"
Oct 30 22:24:35.700539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.700519    5510 scope.go:122] "RemoveContainer" containerID="e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"
Oct 30 22:24:35.700806 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:35.700773    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915\": container with ID starting with e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915 not found: ID does not exist" containerID="e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"
Oct 30 22:24:35.700885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.700817    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915"} err="failed to get container status \"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915\": rpc error: code = NotFound desc = could not find container \"e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915\": container with ID starting with e7a9c5aa033be3baf8cbf607f4e009c9714d91bafca619b7d6c5fd712a795915 not found: ID does not exist"
Oct 30 22:24:35.833639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.833598    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:24:35.880314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.880272    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-edf6d5-786df9fcdb-zpccs"]
Oct 30 22:24:35.941156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.941081    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:24:35.943061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:35.943042    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-edf6d5-7564957fc5-zrxfj"]
Oct 30 22:24:37.124710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:37.124675    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b326b3f-8697-46f6-ace0-e167448c9a0d" path="/var/lib/kubelet/pods/6b326b3f-8697-46f6-ace0-e167448c9a0d/volumes"
Oct 30 22:24:37.125234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:37.125101    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87429923-6520-4079-9c03-88d308793ffa" path="/var/lib/kubelet/pods/87429923-6520-4079-9c03-88d308793ffa/volumes"
Oct 30 22:24:38.824830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:38.824773    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:39.036478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:39.036444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.279678 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:43.279643    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:24:43.408380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:43.408339    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:43.737534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:43.737498    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:24:43.951523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:43.951486    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.045533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:45.045496    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.256150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:45.256115    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.425920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:45.425824    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:46.017003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:46.016969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.426776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:50.426738    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:50.919014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:50.918981    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 22:24:51.204554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:51.204476    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:51.353265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:51.353231    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.141484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:53.141443    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.298646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:53.298611    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:53.475451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:53.475424    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:55.556601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:55.556558    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:55.919584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:55.919494    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing" containerID="cri-o://2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5" gracePeriod=25
Oct 30 22:24:57.906470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:57.906434    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:24:58.759186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:58.759127    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main" containerID="cri-o://98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6" gracePeriod=45
Oct 30 22:24:59.018339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.018276    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:24:59.068001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.067976    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068070    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068119    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068150    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068198    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\" (UniqueName: \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068313    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.068379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068360    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\" (UniqueName: \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\") pod \"344fc2e6-dda8-4078-a78e-1e9c66247915\" (UID: \"344fc2e6-dda8-4078-a78e-1e9c66247915\") "
Oct 30 22:24:59.068492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068385    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.068492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068425    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.068739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068717    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.068739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068738    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.068950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.068747    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.070274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.070248    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "kube-api-access-ztjp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.070381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.070280    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:59.070381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.070366    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs" pod "344fc2e6-dda8-4078-a78e-1e9c66247915" (UID: "344fc2e6-dda8-4078-a78e-1e9c66247915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.242762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.242675    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjp9\" (UniqueName: \"kubernetes.io/projected/344fc2e6-dda8-4078-a78e-1e9c66247915-kube-api-access-ztjp9\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.242762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.242716    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344fc2e6-dda8-4078-a78e-1e9c66247915-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.242762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.242733    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344fc2e6-dda8-4078-a78e-1e9c66247915-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.866878    5510 generic.go:356] "Generic (PLEG): container finished" podID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerID="98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6" exitCode=0
Oct 30 22:24:59.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.867103    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" event={"ID":"344fc2e6-dda8-4078-a78e-1e9c66247915","Type":"ContainerDied","Data":"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"}
Oct 30 22:24:59.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.867141    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9" event={"ID":"344fc2e6-dda8-4078-a78e-1e9c66247915","Type":"ContainerDied","Data":"59651daad7d0f6fa87852c25b273eee6b6c9fd182fdeb68b65a01c50b9d92575"}
Oct 30 22:24:59.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.867166    5510 scope.go:122] "RemoveContainer" containerID="98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"
Oct 30 22:24:59.867805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.867312    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"
Oct 30 22:24:59.947284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.946931    5510 scope.go:122] "RemoveContainer" containerID="98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"
Oct 30 22:24:59.947686 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:24:59.947569    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6\": container with ID starting with 98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6 not found: ID does not exist" containerID="98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"
Oct 30 22:24:59.947686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:24:59.947616    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6"} err="failed to get container status \"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6\": rpc error: code = NotFound desc = could not find container \"98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6\": container with ID starting with 98a08b1da5acc443e9b2639861b7f0556a8b79fbd89ed9eb339113c24a2b76b6 not found: ID does not exist"
Oct 30 22:25:00.027518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:00.027463    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:25:00.093806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:00.086502    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-17d809/distinct-2-w4-17d809-kserve-554c4cf46f-5bjx9"]
Oct 30 22:25:00.596802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:00.596336    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:01.134810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:01.134570    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" path="/var/lib/kubelet/pods/344fc2e6-dda8-4078-a78e-1e9c66247915/volumes"
Oct 30 22:25:01.903989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:01.903945    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:05.456122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:05.456071    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:07.675160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:07.675122    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:08.491500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:08.491465    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:09.811463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:09.811429    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:11.338723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:11.338693    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:11.575048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:11.575013    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:11.732726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:11.732685    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:11.903887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:11.903847    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:12.244655 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:12.244625    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:13.280894 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:13.280849    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:25:16.178415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.178354    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 22:25:16.257389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.257356    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\" (UniqueName: \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\") pod \"5a619273-e44e-4876-bb4a-865f68eaa0f3\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") "
Oct 30 22:25:16.257561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.257416    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\") pod \"5a619273-e44e-4876-bb4a-865f68eaa0f3\" (UID: \"5a619273-e44e-4876-bb4a-865f68eaa0f3\") "
Oct 30 22:25:16.257953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.257803    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume" pod "5a619273-e44e-4876-bb4a-865f68eaa0f3" (UID: "5a619273-e44e-4876-bb4a-865f68eaa0f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:16.259457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.259432    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z" pod "5a619273-e44e-4876-bb4a-865f68eaa0f3" (UID: "5a619273-e44e-4876-bb4a-865f68eaa0f3"). InnerVolumeSpecName "kube-api-access-67f9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:16.359039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.359006    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a619273-e44e-4876-bb4a-865f68eaa0f3-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.359039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.359033    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67f9z\" (UniqueName: \"kubernetes.io/projected/5a619273-e44e-4876-bb4a-865f68eaa0f3-kube-api-access-67f9z\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.907267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:16.907233    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:17.014571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.014534    5510 generic.go:356] "Generic (PLEG): container finished" podID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerID="2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5" exitCode=0
Oct 30 22:25:17.014739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.014596    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"
Oct 30 22:25:17.014739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.014618    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" event={"ID":"5a619273-e44e-4876-bb4a-865f68eaa0f3","Type":"ContainerDied","Data":"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"}
Oct 30 22:25:17.014739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.014698    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm" event={"ID":"5a619273-e44e-4876-bb4a-865f68eaa0f3","Type":"ContainerDied","Data":"6599640e808141634c3853a139ea2996e2c37aa5800f3f4190146dcdcc365787"}
Oct 30 22:25:17.014739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.014721    5510 scope.go:122] "RemoveContainer" containerID="2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"
Oct 30 22:25:17.024874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.024857    5510 scope.go:122] "RemoveContainer" containerID="2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"
Oct 30 22:25:17.025132 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:17.025113    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5\": container with ID starting with 2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5 not found: ID does not exist" containerID="2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"
Oct 30 22:25:17.025191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.025139    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5"} err="failed to get container status \"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5\": rpc error: code = NotFound desc = could not find container \"2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5\": container with ID starting with 2338f8bbd94895ba0b104201dcee4dcf2af27f73e40a1a5160038ff7daf5f2a5 not found: ID does not exist"
Oct 30 22:25:17.187746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.187668    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 22:25:17.233661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:17.233628    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-17d809-769b5db67-7s8sm"]
Oct 30 22:25:19.124965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:19.124931    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" path="/var/lib/kubelet/pods/5a619273-e44e-4876-bb4a-865f68eaa0f3/volumes"
Oct 30 22:25:19.319479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:19.319447    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:19.478858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:19.478814    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:19.679764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:19.679730    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:19.854747 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:19.854659    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:20.514742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:20.514706    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:20.695196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:20.695106    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:21.127681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:21.127650    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.624498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:22.624455    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.873349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:22.873318    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.024789 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:23.024752    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.442369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:23.442278    5510 scope.go:122] "RemoveContainer" containerID="8873fed740a48b57ac668387534f78bad4388c7aa83a1ccbff0655dd107b063f"
Oct 30 22:25:23.451234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:23.451215    5510 scope.go:122] "RemoveContainer" containerID="2b97369ec5d29fc17612cb8773b196ff9b9c433d896728df37f366ad0a5ccd1f"
Oct 30 22:25:23.459420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:23.459394    5510 scope.go:122] "RemoveContainer" containerID="d8863212402a375e3f53938c6c6fa4fae676d95c3691ed7658b573583a38e43e"
Oct 30 22:25:23.576289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:23.576256    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:28.141530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:28.141488    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.061230 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:30.061197    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.217674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:30.217639    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:30.455390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:30.455297    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:25:30.455587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:30.455558    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" containerID="cri-o://d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca" gracePeriod=30
Oct 30 22:25:30.477508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:30.477480    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:31.411698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:31.411669    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:32.333003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.332969    5510 patch_prober.go:29] interesting pod/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:32.333179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.333023    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:32.920000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.919968    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:32.920504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920481    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920496    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920510    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920517    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920526    5510 cpu_manager.go:414] "containerMap: removing container" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920531    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920539    5510 cpu_manager.go:414] "containerMap: removing container" podUID="87429923-6520-4079-9c03-88d308793ffa" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920544    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87429923-6520-4079-9c03-88d308793ffa" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920558    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b902d615-4402-4008-95ee-b048ca9be731" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920562    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b902d615-4402-4008-95ee-b048ca9be731" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920570    5510 cpu_manager.go:414] "containerMap: removing container" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920575    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920591    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6b326b3f-8697-46f6-ace0-e167448c9a0d" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920597    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b326b3f-8697-46f6-ace0-e167448c9a0d" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920610    5510 cpu_manager.go:414] "containerMap: removing container" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920615    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920626    5510 cpu_manager.go:414] "containerMap: removing container" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920631    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920639    5510 cpu_manager.go:414] "containerMap: removing container" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920670    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920740    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a619273-e44e-4876-bb4a-865f68eaa0f3" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920749    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="127344d8-b678-4c8e-935d-36d812f108e5" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920757    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdd14ec-7908-4321-848c-c51cf2f14e7f" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920767    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920774    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879e9b5-fece-49a0-a747-e56664ef2df4" containerName="istio-proxy"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920794    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="344fc2e6-dda8-4078-a78e-1e9c66247915" containerName="main"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920799    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="87429923-6520-4079-9c03-88d308793ffa" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920806    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2a7f65-1d47-4584-85b9-238d9a98bedb" containerName="istio-proxy"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920812    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b902d615-4402-4008-95ee-b048ca9be731" containerName="payload-pre-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920819    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="67065ea8-6706-4f52-991f-b7153cf55f6e" containerName="payload-processing"
Oct 30 22:25:32.920813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.920828    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b326b3f-8697-46f6-ace0-e167448c9a0d" containerName="payload-pre-processing"
Oct 30 22:25:32.925808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.925776    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.974744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.974716    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-17d809-openshift-default-dockercfg-ppc95\""
Oct 30 22:25:32.988340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988305    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988461    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988577    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988658    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9pj\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988729    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988862    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988914    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:32.988949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:32.988942    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.024505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.024468    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:33.090490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090448    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.090688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090510    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.090688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090546    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9pj\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.090920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090885    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090972    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.090977    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091020    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091051    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091122    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091213    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091231    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091619    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.091994 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.091973    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.092215 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.092194    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.093388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.093365    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.094296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.094265    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.144570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.144533    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.144766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.144578    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9pj\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\") pod \"e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") " pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.195931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.195856    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:33.245582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.245096    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:33.688861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.688827    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:33.735125 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:25:33.735089    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a610a7_d55f_4d1e_b3c9_2df566d527c0.slice/crio-177f9e18410374066a04b197936a261bf8fc067f9baac04b44558b83a848fc89 WatchSource:0}: Error finding container 177f9e18410374066a04b197936a261bf8fc067f9baac04b44558b83a848fc89: Status 404 returned error can't find the container with id 177f9e18410374066a04b197936a261bf8fc067f9baac04b44558b83a848fc89
Oct 30 22:25:33.737347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.737308    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:25:33.737448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.737396    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:25:33.737448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:33.737440    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:25:34.122575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.122494    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" event={"ID":"90a610a7-d55f-4d1e-b3c9-2df566d527c0","Type":"ContainerStarted","Data":"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"}
Oct 30 22:25:34.122575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.122542    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" event={"ID":"90a610a7-d55f-4d1e-b3c9-2df566d527c0","Type":"ContainerStarted","Data":"177f9e18410374066a04b197936a261bf8fc067f9baac04b44558b83a848fc89"}
Oct 30 22:25:34.245942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.245909    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:34.247597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.247568    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:34.247731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.247624    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:34.288805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:34.288431    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podStartSLOduration=2.288407626 podStartE2EDuration="2.288407626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:34.280255709 +0000 UTC m=+2543.264162587" watchObservedRunningTime="2026-10-30 22:25:34.288407626 +0000 UTC m=+2543.272314505"
Oct 30 22:25:35.245865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.245827    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:35.246288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.245884    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:35.791454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.791414    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:25:35.831487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831457    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831532    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831575    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831598    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831730    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831768    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.831859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831843    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.832036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831887    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.832036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831907    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:35.832036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.831945    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\") pod \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\" (UID: \"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1\") "
Oct 30 22:25:35.832190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832008    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:35.832190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832120    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:35.832347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832325    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:35.832409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832340    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:35.832578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832555    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.832645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832584    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.832645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832600    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.832645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832615    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.832645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.832629    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.833867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.833843    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "kube-api-access-9l6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:35.833981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.833909    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:35.834181 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.834164    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:35.834244 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.834236    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token" pod "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" (UID: "cb9cebe2-8a6f-41c7-b619-c84dc4e243f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:35.934088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.934052    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.934088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.934083    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.934088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.934092    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:35.934340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:35.934102    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l6wh\" (UniqueName: \"kubernetes.io/projected/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1-kube-api-access-9l6wh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:36.138608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.138524    5510 generic.go:356] "Generic (PLEG): container finished" podID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerID="d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca" exitCode=0
Oct 30 22:25:36.138608 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.138568    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"
Oct 30 22:25:36.138875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.138615    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerDied","Data":"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"}
Oct 30 22:25:36.138875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.138667    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8" event={"ID":"cb9cebe2-8a6f-41c7-b619-c84dc4e243f1","Type":"ContainerDied","Data":"81c78d8b0ad646562c34fb9f9976b5cd9bfd0dbe1b7cee809954cb11b7e525bf"}
Oct 30 22:25:36.138875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.138692    5510 scope.go:122] "RemoveContainer" containerID="d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"
Oct 30 22:25:36.150013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.149993    5510 scope.go:122] "RemoveContainer" containerID="4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"
Oct 30 22:25:36.187776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.187440    5510 scope.go:122] "RemoveContainer" containerID="d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"
Oct 30 22:25:36.187932 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:36.187911    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca\": container with ID starting with d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca not found: ID does not exist" containerID="d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"
Oct 30 22:25:36.188002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.187968    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca"} err="failed to get container status \"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca\": rpc error: code = NotFound desc = could not find container \"d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca\": container with ID starting with d16037ef730b84c10f2a6f00d751c21e45554b63f7c686ed9b18d7747af2b0ca not found: ID does not exist"
Oct 30 22:25:36.188002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.187995    5510 scope.go:122] "RemoveContainer" containerID="4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"
Oct 30 22:25:36.188336 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:36.188301    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002\": container with ID starting with 4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002 not found: ID does not exist" containerID="4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"
Oct 30 22:25:36.188423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.188340    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002"} err="failed to get container status \"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002\": rpc error: code = NotFound desc = could not find container \"4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002\": container with ID starting with 4944cd0760f4a285408a6725b7f1867c3569a96f2c7bdb68dfab31d32c8e9002 not found: ID does not exist"
Oct 30 22:25:36.245826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.245795    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:36.245952 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.245842    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:36.293898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.293860    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:25:36.298606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:36.298576    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-edf6d5-openshift-default-785899c849-fgzk8"]
Oct 30 22:25:37.124861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:37.124830    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" path="/var/lib/kubelet/pods/cb9cebe2-8a6f-41c7-b619-c84dc4e243f1/volumes"
Oct 30 22:25:37.245808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:37.245750    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:37.246011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:37.245832    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:38.246196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:38.246163    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:38.246586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:38.246214    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:38.288153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:38.288081    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:38.932190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:38.932155    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:25:39.246456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:39.246423    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:39.246892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:39.246485    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:40.245691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:40.245656    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 22:25:40.245999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:40.245713    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 22:25:41.249856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:41.249826    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:41.250349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:41.250296    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:41.251033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:41.251002    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:42.438805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:42.438744    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:42.729607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:42.729529    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:25:43.107707 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:43.107631    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.286213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:43.286127    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:43.330827 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:43.330020    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:25:43.953706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:43.953672    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.198224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:44.198191    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.456030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:44.455991    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:44.971394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:44.971363    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:47.730949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:47.730889    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" podUID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" containerName="payload-processing" containerID="cri-o://47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301" gracePeriod=25
Oct 30 22:25:47.989123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:47.989066    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:25:48.015358 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.015326    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\" (UniqueName: \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\") pod \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") "
Oct 30 22:25:48.015514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.015500    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\") pod \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\" (UID: \"6e6f3fb4-d028-4c27-a5c3-5871fe202278\") "
Oct 30 22:25:48.015944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.015918    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume" pod "6e6f3fb4-d028-4c27-a5c3-5871fe202278" (UID: "6e6f3fb4-d028-4c27-a5c3-5871fe202278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:48.017313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.017292    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n" pod "6e6f3fb4-d028-4c27-a5c3-5871fe202278" (UID: "6e6f3fb4-d028-4c27-a5c3-5871fe202278"). InnerVolumeSpecName "kube-api-access-5456n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:48.116708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.116671    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6f3fb4-d028-4c27-a5c3-5871fe202278-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.116708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.116702    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5456n\" (UniqueName: \"kubernetes.io/projected/6e6f3fb4-d028-4c27-a5c3-5871fe202278-kube-api-access-5456n\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:48.239242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.239154    5510 generic.go:356] "Generic (PLEG): container finished" podID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" containerID="47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301" exitCode=0
Oct 30 22:25:48.239242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.239221    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"
Oct 30 22:25:48.239485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.239292    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" event={"ID":"6e6f3fb4-d028-4c27-a5c3-5871fe202278","Type":"ContainerDied","Data":"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"}
Oct 30 22:25:48.239485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.239340    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp" event={"ID":"6e6f3fb4-d028-4c27-a5c3-5871fe202278","Type":"ContainerDied","Data":"4c9fbcac6368b743d24ddee44dcd43fc627fb88431a8eea724b75e658cc1ef63"}
Oct 30 22:25:48.239485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.239368    5510 scope.go:122] "RemoveContainer" containerID="47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"
Oct 30 22:25:48.252324 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.252304    5510 scope.go:122] "RemoveContainer" containerID="47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"
Oct 30 22:25:48.252619 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:48.252598    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301\": container with ID starting with 47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301 not found: ID does not exist" containerID="47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"
Oct 30 22:25:48.252700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.252632    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301"} err="failed to get container status \"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301\": rpc error: code = NotFound desc = could not find container \"47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301\": container with ID starting with 47a37141cc856a23f6f1b404d2c213b4c3e1755aaffd7a307b667518eb3e1301 not found: ID does not exist"
Oct 30 22:25:48.393297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.393263    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:25:48.397145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:48.397118    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c1e2ab-787cf9c97b-jn8qp"]
Oct 30 22:25:49.125529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:49.125493    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" path="/var/lib/kubelet/pods/6e6f3fb4-d028-4c27-a5c3-5871fe202278/volumes"
Oct 30 22:25:49.406410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:49.406323    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.142929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:51.142897    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.308908 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:51.308876    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:51.479928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:51.479891    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:51.480146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:51.480111    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" containerID="cri-o://82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047" gracePeriod=30
Oct 30 22:25:51.511586 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:51.511550    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.163974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:54.163937    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.252422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.252387    5510 patch_prober.go:29] interesting pod/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:56.252822 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.252463    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:56.345686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.345653    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.740004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.739974    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:56.849416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849331    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849492    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849532    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849584    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849702    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849773    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.849851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849832    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.850043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849879    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\") pod \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\" (UID: \"90a610a7-d55f-4d1e-b3c9-2df566d527c0\") "
Oct 30 22:25:56.850043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.849874    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.850043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.850219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850044    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:56.850219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850144    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.850323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850294    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.850588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850564    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.850678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850593    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.850678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850608    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.850678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850621    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.850678 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.850637    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.851718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.851693    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:56.851866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.851724    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:56.851866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.851822    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:56.852188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.852171    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj" pod "90a610a7-d55f-4d1e-b3c9-2df566d527c0" (UID: "90a610a7-d55f-4d1e-b3c9-2df566d527c0"). InnerVolumeSpecName "kube-api-access-cs9pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:56.951607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.951575    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.951607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.951603    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9pj\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-kube-api-access-cs9pj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.951607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.951613    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:56.952008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:56.951622    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a610a7-d55f-4d1e-b3c9-2df566d527c0-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:57.231056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.231024    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.308455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.308425    5510 generic.go:356] "Generic (PLEG): container finished" podID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerID="82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047" exitCode=0
Oct 30 22:25:57.308971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.308481    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"
Oct 30 22:25:57.308971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.308508    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" event={"ID":"90a610a7-d55f-4d1e-b3c9-2df566d527c0","Type":"ContainerDied","Data":"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"}
Oct 30 22:25:57.308971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.308575    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw" event={"ID":"90a610a7-d55f-4d1e-b3c9-2df566d527c0","Type":"ContainerDied","Data":"177f9e18410374066a04b197936a261bf8fc067f9baac04b44558b83a848fc89"}
Oct 30 22:25:57.308971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.308610    5510 scope.go:122] "RemoveContainer" containerID="82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"
Oct 30 22:25:57.318140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.318120    5510 scope.go:122] "RemoveContainer" containerID="82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"
Oct 30 22:25:57.318451 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:25:57.318412    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047\": container with ID starting with 82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047 not found: ID does not exist" containerID="82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"
Oct 30 22:25:57.318531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.318456    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047"} err="failed to get container status \"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047\": rpc error: code = NotFound desc = could not find container \"82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047\": container with ID starting with 82d927c352503bc1c0ab9581bbb41ca6973bb3973b9de934a97b9db5a4399047 not found: ID does not exist"
Oct 30 22:25:57.461488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.461460    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:57.507301 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:57.507265    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-17d809-openshift-default-5b676976ff-b82rw"]
Oct 30 22:25:58.983559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:58.983524    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:59.126525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:25:59.126493    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" path="/var/lib/kubelet/pods/90a610a7-d55f-4d1e-b3c9-2df566d527c0/volumes"
Oct 30 22:26:03.897144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:03.897066    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.660588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:08.660544    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:10.998578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:10.998537    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:26:13.279719 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:26:13.279682    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:26:13.521640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:13.521568    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:18.266229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:18.266195    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.087251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:23.087212    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:27.898502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:27.898456    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:31.721924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:31.721890    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:32.757072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:32.757042    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.553177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:37.553141    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:39.394651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:39.394613    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:39.402884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:39.402859    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:26:42.460146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:42.460111    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:42.972184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:42.972157    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.134195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:43.134156    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:43.280474 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:26:43.280378    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:26:47.260289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:47.260257    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:51.334545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.334469    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:26:51.335326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.334679    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" podUID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" containerName="maas-api" containerID="cri-o://25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444" gracePeriod=30
Oct 30 22:26:51.465458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.465413    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:51.624904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.624813    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:26:51.637976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.637947    5510 generic.go:356] "Generic (PLEG): container finished" podID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" containerID="25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444" exitCode=0
Oct 30 22:26:51.638142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.637981    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" event={"ID":"1f980b9b-c2b1-47be-aad4-f1b0ce244d13","Type":"ContainerDied","Data":"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"}
Oct 30 22:26:51.638142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.638008    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"
Oct 30 22:26:51.638142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.638051    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb" event={"ID":"1f980b9b-c2b1-47be-aad4-f1b0ce244d13","Type":"ContainerDied","Data":"cc97cc639a87db49858a3099a4ed206236d4e4c6bacd74850d34765893387980"}
Oct 30 22:26:51.638142 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.638074    5510 scope.go:122] "RemoveContainer" containerID="25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"
Oct 30 22:26:51.647398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.647380    5510 scope.go:122] "RemoveContainer" containerID="25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"
Oct 30 22:26:51.647702 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:26:51.647674    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444\": container with ID starting with 25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444 not found: ID does not exist" containerID="25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"
Oct 30 22:26:51.647809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.647711    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444"} err="failed to get container status \"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444\": rpc error: code = NotFound desc = could not find container \"25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444\": container with ID starting with 25b0657024e88c79f62d2fb8605ab21a7ad734b42df22e85aa0a15af59207444 not found: ID does not exist"
Oct 30 22:26:51.791156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.791119    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\") pod \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") "
Oct 30 22:26:51.792896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.792868    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") pod \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") "
Oct 30 22:26:51.793444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.793425    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\") pod \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") "
Oct 30 22:26:51.793662 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.793648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\" (UniqueName: \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\") pod \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\" (UID: \"1f980b9b-c2b1-47be-aad4-f1b0ce244d13\") "
Oct 30 22:26:51.795153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.795118    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca" pod "1f980b9b-c2b1-47be-aad4-f1b0ce244d13" (UID: "1f980b9b-c2b1-47be-aad4-f1b0ce244d13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:51.800418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.799767    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls" pod "1f980b9b-c2b1-47be-aad4-f1b0ce244d13" (UID: "1f980b9b-c2b1-47be-aad4-f1b0ce244d13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:51.800418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.799983    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl" pod "1f980b9b-c2b1-47be-aad4-f1b0ce244d13" (UID: "1f980b9b-c2b1-47be-aad4-f1b0ce244d13"). InnerVolumeSpecName "kube-api-access-mfrcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:51.800418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.800076    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs" pod "1f980b9b-c2b1-47be-aad4-f1b0ce244d13" (UID: "1f980b9b-c2b1-47be-aad4-f1b0ce244d13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:51.897620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.897533    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:51.897861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.897842    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:51.897998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.897975    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:51.898097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:51.898084    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfrcl\" (UniqueName: \"kubernetes.io/projected/1f980b9b-c2b1-47be-aad4-f1b0ce244d13-kube-api-access-mfrcl\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:52.051319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:52.051282    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:26:52.056323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:52.056291    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c1e2ab-fc59ddbb6-h4qwb"]
Oct 30 22:26:53.013954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:53.013916    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.124257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:53.124226    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" path="/var/lib/kubelet/pods/1f980b9b-c2b1-47be-aad4-f1b0ce244d13/volumes"
Oct 30 22:26:53.256647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:53.256595    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.420295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:53.420180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.925671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:53.925631    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.467866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:26:58.467834    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:00.226933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:00.226898    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:00.452248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:00.452201    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:00.560483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:00.560402    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:27:00.560703 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:00.560659    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy" containerID="cri-o://7be10dd101a6d5c5144c70df423254cb5d52e6210b7a49224d9962eb9ea37fe4" gracePeriod=30
Oct 30 22:27:00.674514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:00.674420    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:01.089241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:01.089211    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:03.370600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:03.370570    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:05.726056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.726024    5510 generic.go:356] "Generic (PLEG): container finished" podID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerID="7be10dd101a6d5c5144c70df423254cb5d52e6210b7a49224d9962eb9ea37fe4" exitCode=0
Oct 30 22:27:05.726489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.726096    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerDied","Data":"7be10dd101a6d5c5144c70df423254cb5d52e6210b7a49224d9962eb9ea37fe4"}
Oct 30 22:27:05.726489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.726167    5510 scope.go:122] "RemoveContainer" containerID="ef988c80068dd083c8ff21f475d59b2a0e9d6df4f296df636a4ac51d5cd0f45f"
Oct 30 22:27:05.834870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.834806    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:27:05.959794 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.959746    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.959894    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.959941    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.959970    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960038    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960075    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960108    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960156    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960221    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\") pod \"c7a26591-e9d7-4d21-8f4f-431a79306b87\" (UID: \"c7a26591-e9d7-4d21-8f4f-431a79306b87\") "
Oct 30 22:27:05.960420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960281    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.960420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960409    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.960589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960345    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:05.960647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960595    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.960647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960611    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.960826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960756    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.960911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960843    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.960911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960860    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.960911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960875    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a26591-e9d7-4d21-8f4f-431a79306b87-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.960911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.960890    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:05.962335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.962300    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "kube-api-access-chv9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:05.962720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.962698    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:05.963014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.962986    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:05.963112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:05.963040    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token" pod "c7a26591-e9d7-4d21-8f4f-431a79306b87" (UID: "c7a26591-e9d7-4d21-8f4f-431a79306b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:06.061826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.061764    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chv9p\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-kube-api-access-chv9p\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.061826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.061820    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.062030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.061836    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.062030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.061849    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a26591-e9d7-4d21-8f4f-431a79306b87-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:06.230834 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.230775    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.733222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.733194    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"
Oct 30 22:27:06.733702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.733186    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b" event={"ID":"c7a26591-e9d7-4d21-8f4f-431a79306b87","Type":"ContainerDied","Data":"042e5d79bb4c3ea86fc383613b5f575154a0919989ec7a75d15b11463dfcc79b"}
Oct 30 22:27:06.733702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.733340    5510 scope.go:122] "RemoveContainer" containerID="7be10dd101a6d5c5144c70df423254cb5d52e6210b7a49224d9962eb9ea37fe4"
Oct 30 22:27:06.737882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.737851    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:27:06.887708 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.887677    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:27:06.895016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:06.894985    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c1e2ab-openshift-default-bcd8478cf-6c68b"]
Oct 30 22:27:07.124767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:07.124689    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" path="/var/lib/kubelet/pods/c7a26591-e9d7-4d21-8f4f-431a79306b87/volumes"
Oct 30 22:27:08.702577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:08.702547    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.152805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:13.152751    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 22:27:13.284791 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:27:13.284732    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:27:13.410928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:13.410841    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.559451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:13.559395    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.154083 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:18.154045    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing" containerID="cri-o://5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb" gracePeriod=25
Oct 30 22:27:18.269269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:18.269233    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:20.643517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:20.643475    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:23.038480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:23.038442    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.965157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:27.965119    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:29.370635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:29.370602    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:27:30.644226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:30.644180    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:27:30.741106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:30.741070    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:32.055501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:32.055460    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:32.800944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:32.800906    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:37.864100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:37.864061    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:38.405902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.405879    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 22:27:38.428893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.428827    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\") pod \"c528264e-85f1-4100-9cfb-36393cd6517b\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") "
Oct 30 22:27:38.428893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.428881    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\" (UniqueName: \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\") pod \"c528264e-85f1-4100-9cfb-36393cd6517b\" (UID: \"c528264e-85f1-4100-9cfb-36393cd6517b\") "
Oct 30 22:27:38.429116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.429096    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume" pod "c528264e-85f1-4100-9cfb-36393cd6517b" (UID: "c528264e-85f1-4100-9cfb-36393cd6517b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:38.429207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.429188    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c528264e-85f1-4100-9cfb-36393cd6517b-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:38.430845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.430820    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh" pod "c528264e-85f1-4100-9cfb-36393cd6517b" (UID: "c528264e-85f1-4100-9cfb-36393cd6517b"). InnerVolumeSpecName "kube-api-access-djtfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:38.530766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.530737    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtfh\" (UniqueName: \"kubernetes.io/projected/c528264e-85f1-4100-9cfb-36393cd6517b-kube-api-access-djtfh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:38.931394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.931355    5510 generic.go:356] "Generic (PLEG): container finished" podID="c528264e-85f1-4100-9cfb-36393cd6517b" containerID="5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb" exitCode=0
Oct 30 22:27:38.931907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.931401    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"
Oct 30 22:27:38.931907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.931423    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" event={"ID":"c528264e-85f1-4100-9cfb-36393cd6517b","Type":"ContainerDied","Data":"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"}
Oct 30 22:27:38.931907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.931475    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx" event={"ID":"c528264e-85f1-4100-9cfb-36393cd6517b","Type":"ContainerDied","Data":"7dc8e6105c297cc0cc8746a92808b820f39149d42ba0b1280ba5629a41822625"}
Oct 30 22:27:38.931907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.931507    5510 scope.go:122] "RemoveContainer" containerID="5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"
Oct 30 22:27:38.940628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.940613    5510 scope.go:122] "RemoveContainer" containerID="5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"
Oct 30 22:27:38.940854 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:27:38.940836    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb\": container with ID starting with 5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb not found: ID does not exist" containerID="5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"
Oct 30 22:27:38.940914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:38.940864    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb"} err="failed to get container status \"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb\": rpc error: code = NotFound desc = could not find container \"5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb\": container with ID starting with 5bc63b0ef396962437da8fbd59242a7adc0484363e4f928e534977640806cfcb not found: ID does not exist"
Oct 30 22:27:39.084291 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:39.084258    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 22:27:39.130074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:39.130048    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-14af5c-58fd57b68c-5t4nx"]
Oct 30 22:27:41.124226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:41.124193    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" path="/var/lib/kubelet/pods/c528264e-85f1-4100-9cfb-36393cd6517b/volumes"
Oct 30 22:27:42.678123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:42.678066    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.281194 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:27:43.281162    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:27:43.782839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:43.782753    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:43.973119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:43.973087    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:44.340175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:44.340145    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:47.486483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:47.486444    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:48.422210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:48.422177    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:27:50.741434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:50.741406    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:50.895019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:50.894984    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:52.294615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:52.294581    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:57.167723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:27:57.167685    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:01.906677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:01.906643    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:02.070354 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:02.070323    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:06.760326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:06.760287    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.011658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:11.011618    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.878431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:11.878404    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:28:12.332623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:12.332596    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:28:12.590523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:12.590441    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.810116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:12.810084    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:12.978259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:12.978221    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:13.280066 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:28:13.279977    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:28:13.500132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:13.500097    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:17.880139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:17.880100    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.687261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:18.687223    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:28:19.931468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:19.931426    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.144626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:20.144593    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.299816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:20.299738    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:22.790985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:22.790948    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:27.601707 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:27.601661    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.360695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:32.360663    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.527452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:32.527420    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:37.160919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:37.160884    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:39.709942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:39.709910    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:41.981966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:41.981926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:43.280370 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:28:43.280327    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:28:46.758606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:46.758572    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:51.588578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:51.588540    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:53.021091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:53.021053    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:56.439237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:56.439206    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:59.651863 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:28:59.651830    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:29:01.273037 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:01.273003    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.161620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:06.161581    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:10.950940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:10.950904    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:13.279665 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:29:13.279617    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:29:14.692343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:14.692305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.732530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:15.732496    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:29:15.883308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:15.883231    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:21.006375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:21.006342    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.845623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:25.845585    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:26.680951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:26.680920    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:29:27.076417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:27.076380    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:27.252550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:27.252098    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:29.875506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:29.875472    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:29:30.668220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:30.668190    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:31.682159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:31.682115    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing" containerID="cri-o://5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007" gracePeriod=25
Oct 30 22:29:32.735056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:32.735009    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.822120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:34.822086    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.984809 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:34.984115    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:35.487150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:35.487117    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:37.303508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:37.303471    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:40.261028 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:40.260991    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.280255 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:29:43.280219    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:29:44.448450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:44.448413    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:45.940006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:45.939961    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:46.156838 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:46.156806    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:46.308650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:46.308612    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:46.955527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:46.955495    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:47.303815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:47.303757    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:29:50.498323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:50.498287    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:29:51.258806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:51.258766    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:51.945389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:51.945335    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:29:52.039179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.039149    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\" (UniqueName: \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\") pod \"3526fa1d-5bee-4a4b-9f57-059723e859c2\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") "
Oct 30 22:29:52.039374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.039356    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\") pod \"3526fa1d-5bee-4a4b-9f57-059723e859c2\" (UID: \"3526fa1d-5bee-4a4b-9f57-059723e859c2\") "
Oct 30 22:29:52.039705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.039681    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume" pod "3526fa1d-5bee-4a4b-9f57-059723e859c2" (UID: "3526fa1d-5bee-4a4b-9f57-059723e859c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:52.041212 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.041194    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45" pod "3526fa1d-5bee-4a4b-9f57-059723e859c2" (UID: "3526fa1d-5bee-4a4b-9f57-059723e859c2"). InnerVolumeSpecName "kube-api-access-rfg45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:52.140973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.140940    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3526fa1d-5bee-4a4b-9f57-059723e859c2-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:52.140973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.140967    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfg45\" (UniqueName: \"kubernetes.io/projected/3526fa1d-5bee-4a4b-9f57-059723e859c2-kube-api-access-rfg45\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:52.683366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.683329    5510 generic.go:356] "Generic (PLEG): container finished" podID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerID="5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007" exitCode=0
Oct 30 22:29:52.683577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.683382    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"
Oct 30 22:29:52.683577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.683416    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" event={"ID":"3526fa1d-5bee-4a4b-9f57-059723e859c2","Type":"ContainerDied","Data":"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"}
Oct 30 22:29:52.683577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.683512    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb" event={"ID":"3526fa1d-5bee-4a4b-9f57-059723e859c2","Type":"ContainerDied","Data":"882a23c4b5573ac126262b56dd4eb0f22bb797e5311b95a350d206f114bd1801"}
Oct 30 22:29:52.683577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.683544    5510 scope.go:122] "RemoveContainer" containerID="5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"
Oct 30 22:29:52.694284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.694265    5510 scope.go:122] "RemoveContainer" containerID="5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"
Oct 30 22:29:52.694563 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:29:52.694544    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007\": container with ID starting with 5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007 not found: ID does not exist" containerID="5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"
Oct 30 22:29:52.694609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.694572    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007"} err="failed to get container status \"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007\": rpc error: code = NotFound desc = could not find container \"5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007\": container with ID starting with 5bcf80ad316f43fd724c28fbd8b44f29562844abf381c2731dda94d397020007 not found: ID does not exist"
Oct 30 22:29:52.834509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.834474    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:29:52.880161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.880122    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3aa85e-55d55dcbf4-shrjb"]
Oct 30 22:29:52.903619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:52.903588    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.053211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.053176    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.125404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.125370    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" path="/var/lib/kubelet/pods/3526fa1d-5bee-4a4b-9f57-059723e859c2/volumes"
Oct 30 22:29:53.196640 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.196601    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:29:53.196893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.196866    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy" containerID="cri-o://cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152" gracePeriod=30
Oct 30 22:29:53.258459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.258425    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.470251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:53.470216    5510 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal" podUID="0e90b7cbbf3b5bb50f8f3af03ed08b90" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 22:29:56.092561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:56.092531    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:58.455616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.455590    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:29:58.515583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515522    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515566    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515585    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515710    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515751    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515768    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:58.515957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515807    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.515957 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515931    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.516134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.515944    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:58.516134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516028    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:58.516134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516053    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.516134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516093    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\") pod \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\" (UID: \"083c59b3-fcea-4225-b6a9-e46fdd1f8b67\") "
Oct 30 22:29:58.516314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516143    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:58.516389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516369    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:58.516513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516493    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.516557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516522    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.516557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516539    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.516648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516554    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.516648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.516570    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.517919 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.517899    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:58.518262 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.518239    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "kube-api-access-lxvqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:58.518344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.518274    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:58.518401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.518354    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy" pod "083c59b3-fcea-4225-b6a9-e46fdd1f8b67" (UID: "083c59b3-fcea-4225-b6a9-e46fdd1f8b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:58.617733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.617698    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.617733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.617725    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.617733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.617736    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvqw\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-kube-api-access-lxvqw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.617976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.617746    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c59b3-fcea-4225-b6a9-e46fdd1f8b67-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:58.719198 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.719158    5510 generic.go:356] "Generic (PLEG): container finished" podID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerID="cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152" exitCode=0
Oct 30 22:29:58.719403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.719212    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"
Oct 30 22:29:58.719403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.719247    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerDied","Data":"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"}
Oct 30 22:29:58.719403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.719339    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v" event={"ID":"083c59b3-fcea-4225-b6a9-e46fdd1f8b67","Type":"ContainerDied","Data":"511d6e4f38b63e1d101512524da10018d05236f01932a0fc7a619d9d50c1e98f"}
Oct 30 22:29:58.719403 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.719369    5510 scope.go:122] "RemoveContainer" containerID="cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"
Oct 30 22:29:58.730305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.730288    5510 scope.go:122] "RemoveContainer" containerID="cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"
Oct 30 22:29:58.739638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.739620    5510 scope.go:122] "RemoveContainer" containerID="cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"
Oct 30 22:29:58.739921 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:29:58.739901    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152\": container with ID starting with cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152 not found: ID does not exist" containerID="cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"
Oct 30 22:29:58.740049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.739929    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152"} err="failed to get container status \"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152\": rpc error: code = NotFound desc = could not find container \"cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152\": container with ID starting with cbd5704d6ca2556f724674cf1462ce414cc11fba4311920f0de8445b35918152 not found: ID does not exist"
Oct 30 22:29:58.740049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.739949    5510 scope.go:122] "RemoveContainer" containerID="cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"
Oct 30 22:29:58.740232 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:29:58.740210    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be\": container with ID starting with cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be not found: ID does not exist" containerID="cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"
Oct 30 22:29:58.740290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.740241    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be"} err="failed to get container status \"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be\": rpc error: code = NotFound desc = could not find container \"cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be\": container with ID starting with cefab519de992896432d31360a9c53b2b631814b9e980ca4bee0b75d35fae6be not found: ID does not exist"
Oct 30 22:29:58.873309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.873235    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:29:58.880094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:58.880063    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3aa85e-openshift-default-67d756b665-nbh4v"]
Oct 30 22:29:59.124205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:29:59.124132    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" path="/var/lib/kubelet/pods/083c59b3-fcea-4225-b6a9-e46fdd1f8b67/volumes"
Oct 30 22:30:00.170257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170217    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:00.170924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170901    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170924    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170954    5510 cpu_manager.go:414] "containerMap: removing container" podUID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" containerName="maas-api"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170962    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" containerName="maas-api"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170974    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.170982    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171000    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171008    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171021    5510 cpu_manager.go:414] "containerMap: removing container" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171029    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171039    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171046    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171058    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" containerName="payload-processing"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171065    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" containerName="payload-processing"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171076    5510 cpu_manager.go:414] "containerMap: removing container" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171085    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171105    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171112    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171138    5510 cpu_manager.go:414] "containerMap: removing container" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171146    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171256    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171272    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3526fa1d-5bee-4a4b-9f57-059723e859c2" containerName="payload-processing"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171285    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f980b9b-c2b1-47be-aad4-f1b0ce244d13" containerName="maas-api"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171293    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c528264e-85f1-4100-9cfb-36393cd6517b" containerName="payload-pre-processing"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171303    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171312    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171320    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6f3fb4-d028-4c27-a5c3-5871fe202278" containerName="payload-processing"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171331    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a610a7-d55f-4d1e-b3c9-2df566d527c0" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171341    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9cebe2-8a6f-41c7-b619-c84dc4e243f1" containerName="istio-proxy"
Oct 30 22:30:00.171617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.171351    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:30:00.176595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.176573    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.224492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.224458    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:30:00.233786 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.233764    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqml\" (UniqueName: \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.233914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.233896    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.272578 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.272549    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:00.335257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.335225    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqml\" (UniqueName: \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.335426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.335315    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.335988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.335967    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.391051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.391021    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqml\" (UniqueName: \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\") pod \"maas-api-key-cleanup-29889990-c6mcc\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.489139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.489104    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:00.860223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:00.860140    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:00.868398 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:30:00.868367    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335da42c_dbf7_4e8e_bef8_21460bdcddf7.slice/crio-2877c0deffeb6805b846771966d7c8f4eb8dc2ad1e0e8712bd9f288c985655f4 WatchSource:0}: Error finding container 2877c0deffeb6805b846771966d7c8f4eb8dc2ad1e0e8712bd9f288c985655f4: Status 404 returned error can't find the container with id 2877c0deffeb6805b846771966d7c8f4eb8dc2ad1e0e8712bd9f288c985655f4
Oct 30 22:30:01.166929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:01.166904    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:01.739161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:01.739122    5510 generic.go:356] "Generic (PLEG): container finished" podID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerID="a506645ff43696c71fed5eac345ac9ce9dd14c34eafae200566133f486b125db" exitCode=60
Oct 30 22:30:01.739605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:01.739208    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerDied","Data":"a506645ff43696c71fed5eac345ac9ce9dd14c34eafae200566133f486b125db"}
Oct 30 22:30:01.739605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:01.739275    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerStarted","Data":"2877c0deffeb6805b846771966d7c8f4eb8dc2ad1e0e8712bd9f288c985655f4"}
Oct 30 22:30:01.739605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:01.739486    5510 scope.go:122] "RemoveContainer" containerID="a506645ff43696c71fed5eac345ac9ce9dd14c34eafae200566133f486b125db"
Oct 30 22:30:02.476890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:02.476860    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:02.747159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:02.747067    5510 generic.go:356] "Generic (PLEG): container finished" podID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerID="d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de" exitCode=60
Oct 30 22:30:02.747605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:02.747153    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerDied","Data":"d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de"}
Oct 30 22:30:02.747605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:02.747217    5510 scope.go:122] "RemoveContainer" containerID="a506645ff43696c71fed5eac345ac9ce9dd14c34eafae200566133f486b125db"
Oct 30 22:30:02.747605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:02.747559    5510 scope.go:122] "RemoveContainer" containerID="d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de"
Oct 30 22:30:02.747922 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:02.747902    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-c6mcc_odh-ai-gateway-infra(335da42c-dbf7-4e8e-bef8-21460bdcddf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7"
Oct 30 22:30:03.754555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:03.754526    5510 scope.go:122] "RemoveContainer" containerID="d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de"
Oct 30 22:30:03.755051 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:03.754882    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-c6mcc_odh-ai-gateway-infra(335da42c-dbf7-4e8e-bef8-21460bdcddf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7"
Oct 30 22:30:05.980115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:05.980082    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:06.151925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:06.150404    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:10.033812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.033766    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:30:10.034321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.033993    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api" containerID="cri-o://799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76" gracePeriod=30
Oct 30 22:30:10.221876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.221841    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:10.300905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.300835    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:30:10.367530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.367497    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\") pod \"02863ada-09c7-4996-9148-4844e22554f2\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") "
Oct 30 22:30:10.367692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.367540    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") pod \"02863ada-09c7-4996-9148-4844e22554f2\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") "
Oct 30 22:30:10.367692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.367573    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\") pod \"02863ada-09c7-4996-9148-4844e22554f2\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") "
Oct 30 22:30:10.367922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.367897    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca" pod "02863ada-09c7-4996-9148-4844e22554f2" (UID: "02863ada-09c7-4996-9148-4844e22554f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:10.368064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.368049    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02863ada-09c7-4996-9148-4844e22554f2-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.369605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.369584    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs" pod "02863ada-09c7-4996-9148-4844e22554f2" (UID: "02863ada-09c7-4996-9148-4844e22554f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:10.369698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.369622    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls" pod "02863ada-09c7-4996-9148-4844e22554f2" (UID: "02863ada-09c7-4996-9148-4844e22554f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:10.468793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.468743    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\" (UniqueName: \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\") pod \"02863ada-09c7-4996-9148-4844e22554f2\" (UID: \"02863ada-09c7-4996-9148-4844e22554f2\") "
Oct 30 22:30:10.469246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.469224    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.469346 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.469251    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02863ada-09c7-4996-9148-4844e22554f2-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.470839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.470811    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt" pod "02863ada-09c7-4996-9148-4844e22554f2" (UID: "02863ada-09c7-4996-9148-4844e22554f2"). InnerVolumeSpecName "kube-api-access-8d2nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:10.570188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.570104    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2nt\" (UniqueName: \"kubernetes.io/projected/02863ada-09c7-4996-9148-4844e22554f2-kube-api-access-8d2nt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:10.790200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.790167    5510 generic.go:356] "Generic (PLEG): container finished" podID="02863ada-09c7-4996-9148-4844e22554f2" containerID="799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76" exitCode=0
Oct 30 22:30:10.790406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.790227    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"
Oct 30 22:30:10.790406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.790254    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerDied","Data":"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"}
Oct 30 22:30:10.790406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.790308    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr" event={"ID":"02863ada-09c7-4996-9148-4844e22554f2","Type":"ContainerDied","Data":"19f32f9745963f2d6460ed2da0ef98e9e6d52eef7b0dd809aedc8a1cc03e177d"}
Oct 30 22:30:10.790406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.790332    5510 scope.go:122] "RemoveContainer" containerID="799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"
Oct 30 22:30:10.800224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.800206    5510 scope.go:122] "RemoveContainer" containerID="447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"
Oct 30 22:30:10.808431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.808409    5510 scope.go:122] "RemoveContainer" containerID="799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"
Oct 30 22:30:10.808721 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:10.808700    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76\": container with ID starting with 799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76 not found: ID does not exist" containerID="799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"
Oct 30 22:30:10.808800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.808729    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76"} err="failed to get container status \"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76\": rpc error: code = NotFound desc = could not find container \"799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76\": container with ID starting with 799198f8a23fc98c7648568bd2bc7b35596f059456462c1d9d93cf4ac969af76 not found: ID does not exist"
Oct 30 22:30:10.808800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.808749    5510 scope.go:122] "RemoveContainer" containerID="447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"
Oct 30 22:30:10.809017 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:10.808999    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1\": container with ID starting with 447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1 not found: ID does not exist" containerID="447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"
Oct 30 22:30:10.809065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.809021    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1"} err="failed to get container status \"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1\": rpc error: code = NotFound desc = could not find container \"447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1\": container with ID starting with 447d0bde488029c39284e749f1ac7b5307b669dd7dad0d8adab960ade25b5ed1 not found: ID does not exist"
Oct 30 22:30:10.943067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.942990    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:30:10.993807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:10.992929    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84a3cf-5cd9db59d-fwnrr"]
Oct 30 22:30:11.124289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:11.124254    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02863ada-09c7-4996-9148-4844e22554f2" path="/var/lib/kubelet/pods/02863ada-09c7-4996-9148-4844e22554f2/volumes"
Oct 30 22:30:11.685668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:11.685637    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:11.981558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:11.981524    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:12.144500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:12.144467    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:12.631173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:12.631139    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:13.281006 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:13.280935    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:30:15.884625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:15.884589    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:30:16.117840 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:16.117809    5510 scope.go:122] "RemoveContainer" containerID="d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de"
Oct 30 22:30:16.824711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:16.824629    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerStarted","Data":"9babe8931a8a46f65b6ce1ef3b315b1f1ec6eed04710c634dcc1d4fb6bfe4f5f"}
Oct 30 22:30:16.932313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:16.932260    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" podStartSLOduration=16.932245867 podStartE2EDuration="16.932245867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:16.932189773 +0000 UTC m=+2825.916096649" watchObservedRunningTime="2026-10-30 22:30:16.932245867 +0000 UTC m=+2825.916152743"
Oct 30 22:30:17.069194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.069166    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:17.832301 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.832266    5510 generic.go:356] "Generic (PLEG): container finished" podID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerID="9babe8931a8a46f65b6ce1ef3b315b1f1ec6eed04710c634dcc1d4fb6bfe4f5f" exitCode=60
Oct 30 22:30:17.832471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.832341    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerDied","Data":"9babe8931a8a46f65b6ce1ef3b315b1f1ec6eed04710c634dcc1d4fb6bfe4f5f"}
Oct 30 22:30:17.832471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.832403    5510 scope.go:122] "RemoveContainer" containerID="d970e24f9cda75e5dbb8f4852f68c2709f9bb2a1fd41538c18be1187119305de"
Oct 30 22:30:17.832793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.832752    5510 scope.go:122] "RemoveContainer" containerID="9babe8931a8a46f65b6ce1ef3b315b1f1ec6eed04710c634dcc1d4fb6bfe4f5f"
Oct 30 22:30:17.833104 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:17.833076    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-c6mcc_odh-ai-gateway-infra(335da42c-dbf7-4e8e-bef8-21460bdcddf7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7"
Oct 30 22:30:17.944753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:17.944722    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:18.882431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:18.882400    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.991406 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:18.991382    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:19.058563 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.058524    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:19.063519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.063496    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\" (UniqueName: \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\") pod \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") "
Oct 30 22:30:19.063730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.063710    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\") pod \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\" (UID: \"335da42c-dbf7-4e8e-bef8-21460bdcddf7\") "
Oct 30 22:30:19.064064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.064039    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca" pod "335da42c-dbf7-4e8e-bef8-21460bdcddf7" (UID: "335da42c-dbf7-4e8e-bef8-21460bdcddf7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.065635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.065611    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml" pod "335da42c-dbf7-4e8e-bef8-21460bdcddf7" (UID: "335da42c-dbf7-4e8e-bef8-21460bdcddf7"). InnerVolumeSpecName "kube-api-access-qcqml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.165018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.164984    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqml\" (UniqueName: \"kubernetes.io/projected/335da42c-dbf7-4e8e-bef8-21460bdcddf7-kube-api-access-qcqml\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.165018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.165017    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335da42c-dbf7-4e8e-bef8-21460bdcddf7-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.265742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.265694    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:19.846203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.846173    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"
Oct 30 22:30:19.846203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.846189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc" event={"ID":"335da42c-dbf7-4e8e-bef8-21460bdcddf7","Type":"ContainerDied","Data":"2877c0deffeb6805b846771966d7c8f4eb8dc2ad1e0e8712bd9f288c985655f4"}
Oct 30 22:30:19.846452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.846247    5510 scope.go:122] "RemoveContainer" containerID="9babe8931a8a46f65b6ce1ef3b315b1f1ec6eed04710c634dcc1d4fb6bfe4f5f"
Oct 30 22:30:19.999996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:19.999963    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:20.045843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.045797    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-c6mcc"]
Oct 30 22:30:20.144827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.144726    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:30:20.144990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.144941    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" containerID="cri-o://2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56" gracePeriod=30
Oct 30 22:30:20.146552 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.146529    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:30:20.146756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.146730    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" containerID="cri-o://0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b" gracePeriod=30
Oct 30 22:30:20.148258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.148226    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:30:20.148432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.148405    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy" containerID="cri-o://6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb" gracePeriod=30
Oct 30 22:30:20.188936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.188914    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:30:20.189078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:20.189061    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" containerID="cri-o://38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df" gracePeriod=30
Oct 30 22:30:21.124285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:21.124249    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" path="/var/lib/kubelet/pods/335da42c-dbf7-4e8e-bef8-21460bdcddf7/volumes"
Oct 30 22:30:21.970027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:21.969991    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:24.822857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:24.822809    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-qvb8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:24.823235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:24.822899    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:24.877131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:24.877094    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-blpfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:30:24.877303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:24.877170    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:30:25.456047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.456018    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:30:25.489412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.489387    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:30:25.528776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.528751    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:30:25.531827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.531805    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:30:25.543162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543143    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543199    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543231    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543256    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543348 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543283    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543348 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543318    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543389    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543440    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543470    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\") pod \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\" (UID: \"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e\") "
Oct 30 22:30:25.543685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543561    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.543685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543615    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.543831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543816    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.543922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.543900    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.544032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.544016    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.544079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.544039    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.544079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.544054    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.544079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.544068    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.544186 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.544017    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:25.545449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.545429    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "kube-api-access-r7544". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.545515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.545451    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:25.545716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.545695    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.545762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.545711    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token" pod "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" (UID: "a21e5add-7f01-4b1c-a8f1-c3071bc83e2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.644964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.644932    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.644985    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645089    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645118    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645145    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.645319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645184    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645231    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645255    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.645319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645288    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645322    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645375    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645409    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645441    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645445    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645472    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645511    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645560    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645608    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.645652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645630    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645659    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645693    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645729    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645763    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645828    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645857    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645879    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645903    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\") pod \"69463924-211a-4117-bb6f-eec4aa88a760\" (UID: \"69463924-211a-4117-bb6f-eec4aa88a760\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645940    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\") pod \"29a1e52b-8886-416a-ac49-e6a6f117559e\" (UID: \"29a1e52b-8886-416a-ac49-e6a6f117559e\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.645970    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.646140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646067    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\") pod \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\" (UID: \"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581\") "
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646358    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646574    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646597    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646614    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646629    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646643    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646657    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646671    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646686    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7544\" (UniqueName: \"kubernetes.io/projected/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-kube-api-access-r7544\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.646721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646700    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.647238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.646958    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.647820    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648042    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648100    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648151    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648417    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648443    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.648547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648440    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.649530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648825    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:25.649530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.648986    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "kube-api-access-mxhcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.649530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.649004    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:25.649530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.649030    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:25.649530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.649271    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.650267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.650228    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.651599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.651154    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:25.651599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.651568    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.652687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.652574    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:25.652687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.652640    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy" pod "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" (UID: "ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.652687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.652662    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z" pod "29a1e52b-8886-416a-ac49-e6a6f117559e" (UID: "29a1e52b-8886-416a-ac49-e6a6f117559e"). InnerVolumeSpecName "kube-api-access-wmt9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.652916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.652757    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.652916 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.652876    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "kube-api-access-pftsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:25.653054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.653037    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:25.653054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.653046    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy" pod "69463924-211a-4117-bb6f-eec4aa88a760" (UID: "69463924-211a-4117-bb6f-eec4aa88a760"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:25.748128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748096    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748123    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748132    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69463924-211a-4117-bb6f-eec4aa88a760-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748142    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748153    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748161    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748169    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748178    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748187    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmt9z\" (UniqueName: \"kubernetes.io/projected/29a1e52b-8886-416a-ac49-e6a6f117559e-kube-api-access-wmt9z\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748198    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748207    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748217    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748225    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748233    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhcv\" (UniqueName: \"kubernetes.io/projected/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-kube-api-access-mxhcv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748243    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a1e52b-8886-416a-ac49-e6a6f117559e-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748252    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748260    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748268    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69463924-211a-4117-bb6f-eec4aa88a760-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748276    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748284    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftsz\" (UniqueName: \"kubernetes.io/projected/69463924-211a-4117-bb6f-eec4aa88a760-kube-api-access-pftsz\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748292    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748302    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a1e52b-8886-416a-ac49-e6a6f117559e-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.748361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.748310    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69463924-211a-4117-bb6f-eec4aa88a760-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:25.886512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.886480    5510 generic.go:356] "Generic (PLEG): container finished" podID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerID="38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df" exitCode=0
Oct 30 22:30:25.886940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.886528    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"
Oct 30 22:30:25.886940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.886569    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" event={"ID":"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581","Type":"ContainerDied","Data":"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"}
Oct 30 22:30:25.886940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.886632    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn" event={"ID":"ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581","Type":"ContainerDied","Data":"a8d2f7a00bdc7544bf66a9a353626982031d785256115d92c867c61879bfcf15"}
Oct 30 22:30:25.886940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.886666    5510 scope.go:122] "RemoveContainer" containerID="38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"
Oct 30 22:30:25.887984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.887964    5510 generic.go:356] "Generic (PLEG): container finished" podID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerID="0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b" exitCode=0
Oct 30 22:30:25.888065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.888016    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"
Oct 30 22:30:25.888122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.888082    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" event={"ID":"29a1e52b-8886-416a-ac49-e6a6f117559e","Type":"ContainerDied","Data":"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"}
Oct 30 22:30:25.888122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.888118    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t" event={"ID":"29a1e52b-8886-416a-ac49-e6a6f117559e","Type":"ContainerDied","Data":"1afb0e11c004f37964f7023b689527dca5fcc344d8b55e7f6742617fc09c3a3d"}
Oct 30 22:30:25.889519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.889498    5510 generic.go:356] "Generic (PLEG): container finished" podID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerID="6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb" exitCode=0
Oct 30 22:30:25.889618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.889556    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"
Oct 30 22:30:25.889618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.889606    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" event={"ID":"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e","Type":"ContainerDied","Data":"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"}
Oct 30 22:30:25.889724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.889640    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796" event={"ID":"a21e5add-7f01-4b1c-a8f1-c3071bc83e2e","Type":"ContainerDied","Data":"c2d29e35b4fb63cf324d6a0a33aa9a52684a7c0be143278f07521054c3d04fc5"}
Oct 30 22:30:25.891049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.891028    5510 generic.go:356] "Generic (PLEG): container finished" podID="69463924-211a-4117-bb6f-eec4aa88a760" containerID="2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56" exitCode=0
Oct 30 22:30:25.891157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.891066    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"
Oct 30 22:30:25.891157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.891069    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" event={"ID":"69463924-211a-4117-bb6f-eec4aa88a760","Type":"ContainerDied","Data":"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"}
Oct 30 22:30:25.891273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.891190    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" event={"ID":"69463924-211a-4117-bb6f-eec4aa88a760","Type":"ContainerDied","Data":"bd7ba8693268665bbfeb7b61e35004b79873ef8fd0137d36c7a43afbac860a97"}
Oct 30 22:30:25.896928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.896908    5510 scope.go:122] "RemoveContainer" containerID="38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"
Oct 30 22:30:25.897194 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:25.897171    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df\": container with ID starting with 38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df not found: ID does not exist" containerID="38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"
Oct 30 22:30:25.897254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.897206    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df"} err="failed to get container status \"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df\": rpc error: code = NotFound desc = could not find container \"38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df\": container with ID starting with 38c13daca169b0eb843e57d1da34b33f1b3efcb103a146d9a6d6d65c029aa7df not found: ID does not exist"
Oct 30 22:30:25.897254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.897229    5510 scope.go:122] "RemoveContainer" containerID="0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"
Oct 30 22:30:25.907492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.907477    5510 scope.go:122] "RemoveContainer" containerID="0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"
Oct 30 22:30:25.907741 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:25.907720    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b\": container with ID starting with 0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b not found: ID does not exist" containerID="0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"
Oct 30 22:30:25.908487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.907746    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b"} err="failed to get container status \"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b\": rpc error: code = NotFound desc = could not find container \"0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b\": container with ID starting with 0fadd226ad9240248f53dfac7ef72db4d4ec6d7829093a55829cb69a2dd27e7b not found: ID does not exist"
Oct 30 22:30:25.908487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.907761    5510 scope.go:122] "RemoveContainer" containerID="6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"
Oct 30 22:30:25.915610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.915594    5510 scope.go:122] "RemoveContainer" containerID="6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"
Oct 30 22:30:25.915860 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:25.915839    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb\": container with ID starting with 6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb not found: ID does not exist" containerID="6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"
Oct 30 22:30:25.915906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.915867    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb"} err="failed to get container status \"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb\": rpc error: code = NotFound desc = could not find container \"6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb\": container with ID starting with 6bf577dff0ace8dd50aaa92bdea3de888c4f33b0318ca8672cae1dcb2d39e9fb not found: ID does not exist"
Oct 30 22:30:25.915906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.915882    5510 scope.go:122] "RemoveContainer" containerID="2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"
Oct 30 22:30:25.923502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.923485    5510 scope.go:122] "RemoveContainer" containerID="2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"
Oct 30 22:30:25.923750 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:25.923732    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56\": container with ID starting with 2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56 not found: ID does not exist" containerID="2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"
Oct 30 22:30:25.923806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:25.923759    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56"} err="failed to get container status \"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56\": rpc error: code = NotFound desc = could not find container \"2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56\": container with ID starting with 2bc98afb751d6c1feec6e42c080e6624ce56b293fd5bfa5759c1820ec3d44e56 not found: ID does not exist"
Oct 30 22:30:26.053391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.053311    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:30:26.059026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.058960    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj"]
Oct 30 22:30:26.159313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.159279    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:30:26.163000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.162968    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-blpfn"]
Oct 30 22:30:26.264973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.264944    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:30:26.267288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.267264    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-qvb8t"]
Oct 30 22:30:26.369063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.368985    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:30:26.373523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.373473    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4c796"]
Oct 30 22:30:26.442007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.441980    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ffhxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:30:26.442189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.442074    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ffhxj" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:26.770530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:26.770494    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:27.125222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:27.125135    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" path="/var/lib/kubelet/pods/29a1e52b-8886-416a-ac49-e6a6f117559e/volumes"
Oct 30 22:30:27.125685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:27.125666    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69463924-211a-4117-bb6f-eec4aa88a760" path="/var/lib/kubelet/pods/69463924-211a-4117-bb6f-eec4aa88a760/volumes"
Oct 30 22:30:27.126265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:27.126245    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" path="/var/lib/kubelet/pods/a21e5add-7f01-4b1c-a8f1-c3071bc83e2e/volumes"
Oct 30 22:30:27.126980 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:27.126961    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" path="/var/lib/kubelet/pods/ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581/volumes"
Oct 30 22:30:31.766410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:31.766374    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.074846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:33.074815    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.861948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:36.861912    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.591059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:41.591014    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:30:41.688626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:41.688593    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:30:41.767942 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:41.767908    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.974489 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:41.974457    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:42.136625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:42.136595    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:43.287185 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:43.287147    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:30:45.898495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:45.898457    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:46.591978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.591910    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" podUID="d5e40124-3293-439d-89d0-082236c776d1" containerName="payload-processing" containerID="cri-o://74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274" gracePeriod=25
Oct 30 22:30:46.689637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.689545    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" podUID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" containerName="payload-pre-processing" containerID="cri-o://e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13" gracePeriod=25
Oct 30 22:30:46.843624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.843556    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:30:46.852012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.851989    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\") pod \"d5e40124-3293-439d-89d0-082236c776d1\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") "
Oct 30 22:30:46.852132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.852056    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\" (UniqueName: \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\") pod \"d5e40124-3293-439d-89d0-082236c776d1\" (UID: \"d5e40124-3293-439d-89d0-082236c776d1\") "
Oct 30 22:30:46.852389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.852364    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume" pod "d5e40124-3293-439d-89d0-082236c776d1" (UID: "d5e40124-3293-439d-89d0-082236c776d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:46.852543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.852520    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e40124-3293-439d-89d0-082236c776d1-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.854068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.854042    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv" pod "d5e40124-3293-439d-89d0-082236c776d1" (UID: "d5e40124-3293-439d-89d0-082236c776d1"). InnerVolumeSpecName "kube-api-access-mw5cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:46.934624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.934596    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:30:46.952943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.952916    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\" (UniqueName: \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\") pod \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") "
Oct 30 22:30:46.953103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.952957    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\") pod \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\" (UID: \"3cfb9742-e8af-4b16-91c8-01e9b08c84b3\") "
Oct 30 22:30:46.953408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.953320    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5cv\" (UniqueName: \"kubernetes.io/projected/d5e40124-3293-439d-89d0-082236c776d1-kube-api-access-mw5cv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:46.953542 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.953439    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume" pod "3cfb9742-e8af-4b16-91c8-01e9b08c84b3" (UID: "3cfb9742-e8af-4b16-91c8-01e9b08c84b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:46.955519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:46.955461    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8" pod "3cfb9742-e8af-4b16-91c8-01e9b08c84b3" (UID: "3cfb9742-e8af-4b16-91c8-01e9b08c84b3"). InnerVolumeSpecName "kube-api-access-7svg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:47.014261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.014226    5510 generic.go:356] "Generic (PLEG): container finished" podID="d5e40124-3293-439d-89d0-082236c776d1" containerID="74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274" exitCode=0
Oct 30 22:30:47.014436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.014276    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"
Oct 30 22:30:47.014436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.014305    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" event={"ID":"d5e40124-3293-439d-89d0-082236c776d1","Type":"ContainerDied","Data":"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"}
Oct 30 22:30:47.014436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.014408    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m" event={"ID":"d5e40124-3293-439d-89d0-082236c776d1","Type":"ContainerDied","Data":"6af371323b3613ae5d120a4d549098d699eebe6f981f1e01ceb95ce41a710b44"}
Oct 30 22:30:47.014639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.014444    5510 scope.go:122] "RemoveContainer" containerID="74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"
Oct 30 22:30:47.015816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.015754    5510 generic.go:356] "Generic (PLEG): container finished" podID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" containerID="e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13" exitCode=0
Oct 30 22:30:47.015940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.015817    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"
Oct 30 22:30:47.015940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.015828    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" event={"ID":"3cfb9742-e8af-4b16-91c8-01e9b08c84b3","Type":"ContainerDied","Data":"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"}
Oct 30 22:30:47.015940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.015875    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d" event={"ID":"3cfb9742-e8af-4b16-91c8-01e9b08c84b3","Type":"ContainerDied","Data":"4c12e33c6a21875e39d72bd5ab9e04b84f600ec3d5abfcdf4ae7aa1597a35fb0"}
Oct 30 22:30:47.025602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.025583    5510 scope.go:122] "RemoveContainer" containerID="74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"
Oct 30 22:30:47.025981 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:47.025960    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274\": container with ID starting with 74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274 not found: ID does not exist" containerID="74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"
Oct 30 22:30:47.026095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.025991    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274"} err="failed to get container status \"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274\": rpc error: code = NotFound desc = could not find container \"74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274\": container with ID starting with 74d96cb8f8603b217cbfcaf03ca6eb9de4530a17a893b19f6c872986833d9274 not found: ID does not exist"
Oct 30 22:30:47.026095 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.026025    5510 scope.go:122] "RemoveContainer" containerID="e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"
Oct 30 22:30:47.035043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.035023    5510 scope.go:122] "RemoveContainer" containerID="e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"
Oct 30 22:30:47.035303 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:30:47.035285    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13\": container with ID starting with e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13 not found: ID does not exist" containerID="e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"
Oct 30 22:30:47.035367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.035311    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13"} err="failed to get container status \"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13\": rpc error: code = NotFound desc = could not find container \"e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13\": container with ID starting with e94dd87032d63fb8aafd24ecce41b3e34cd3bcc16d5eec7d65d1a5d0cad1ae13 not found: ID does not exist"
Oct 30 22:30:47.053965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.053944    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svg8\" (UniqueName: \"kubernetes.io/projected/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-kube-api-access-7svg8\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:47.054060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.053966    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfb9742-e8af-4b16-91c8-01e9b08c84b3-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:47.167618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.167537    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:30:47.172986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.172960    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-4d8041-6986f8c856-bdc8m"]
Oct 30 22:30:47.273135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.273100    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:30:47.279086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.279057    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-4d8041-8588c47659-h7t5d"]
Oct 30 22:30:47.436315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.436240    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.673894 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.673861    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.851025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:47.850994    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:48.360248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:48.360218    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:49.125150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:49.125118    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" path="/var/lib/kubelet/pods/3cfb9742-e8af-4b16-91c8-01e9b08c84b3/volumes"
Oct 30 22:30:49.125496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:49.125483    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e40124-3293-439d-89d0-082236c776d1" path="/var/lib/kubelet/pods/d5e40124-3293-439d-89d0-082236c776d1/volumes"
Oct 30 22:30:51.645462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:51.645424    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:30:52.757433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:52.757394    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.235103 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:54.235070    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.387847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:54.387811    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.608117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:54.608034    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:30:54.608296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:54.608271    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy" containerID="cri-o://4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9" gracePeriod=30
Oct 30 22:30:54.638885 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:54.638858    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:57.810067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:57.810025    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:59.859876 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.859849    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:30:59.977221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977191    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977273    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977312    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977352    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977421    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977454    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977507    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977544    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977561    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977610    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:59.977661 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.977627    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\") pod \"544ea754-1458-4660-939e-4f974fc3cd17\" (UID: \"544ea754-1458-4660-939e-4f974fc3cd17\") "
Oct 30 22:30:59.978273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.978006    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:59.978273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.978061    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/544ea754-1458-4660-939e-4f974fc3cd17-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.978273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.978083    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:59.978273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.978064    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:59.978273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.978086    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:59.979656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.979624    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:59.979762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.979677    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:59.979762 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.979681    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:59.979902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:30:59.979887    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b" pod "544ea754-1458-4660-939e-4f974fc3cd17" (UID: "544ea754-1458-4660-939e-4f974fc3cd17"). InnerVolumeSpecName "kube-api-access-q797b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:00.078828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078771    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.078828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078824    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.079029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078839    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q797b\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-kube-api-access-q797b\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.079029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078852    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/544ea754-1458-4660-939e-4f974fc3cd17-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.079029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078865    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.079029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078876    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/544ea754-1458-4660-939e-4f974fc3cd17-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.079029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.078889    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/544ea754-1458-4660-939e-4f974fc3cd17-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:00.088547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.088514    5510 generic.go:356] "Generic (PLEG): container finished" podID="544ea754-1458-4660-939e-4f974fc3cd17" containerID="4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9" exitCode=0
Oct 30 22:31:00.088698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.088568    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"
Oct 30 22:31:00.088698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.088613    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerDied","Data":"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"}
Oct 30 22:31:00.088698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.088671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz" event={"ID":"544ea754-1458-4660-939e-4f974fc3cd17","Type":"ContainerDied","Data":"9db94a473a818a5fce4599fef338d3748eebdac5e9372b0e7fd1bf43a45fecff"}
Oct 30 22:31:00.088860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.088701    5510 scope.go:122] "RemoveContainer" containerID="4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"
Oct 30 22:31:00.099404 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.098395    5510 scope.go:122] "RemoveContainer" containerID="d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"
Oct 30 22:31:00.106685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.106663    5510 scope.go:122] "RemoveContainer" containerID="4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"
Oct 30 22:31:00.106961 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:00.106942    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9\": container with ID starting with 4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9 not found: ID does not exist" containerID="4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"
Oct 30 22:31:00.107026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.106970    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9"} err="failed to get container status \"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9\": rpc error: code = NotFound desc = could not find container \"4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9\": container with ID starting with 4f7721ccf4830814ea5fbd8dcb8b235e54719eb24bacdf6bed7f34a6220cefc9 not found: ID does not exist"
Oct 30 22:31:00.107026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.106987    5510 scope.go:122] "RemoveContainer" containerID="d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"
Oct 30 22:31:00.107215 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:00.107195    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd\": container with ID starting with d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd not found: ID does not exist" containerID="d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"
Oct 30 22:31:00.107276 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.107224    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd"} err="failed to get container status \"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd\": rpc error: code = NotFound desc = could not find container \"d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd\": container with ID starting with d6f163d59ea485e5755b92580f2d93b82ad3b77fe28444bee513e067ad8385cd not found: ID does not exist"
Oct 30 22:31:00.241641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.241570    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:31:00.245414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:00.245390    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-4d8041-openshift-default-5fcdfd94dc-f4zzz"]
Oct 30 22:31:01.123250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:01.123214    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544ea754-1458-4660-939e-4f974fc3cd17" path="/var/lib/kubelet/pods/544ea754-1458-4660-939e-4f974fc3cd17/volumes"
Oct 30 22:31:02.744988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:02.744955    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:06.990866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:06.990828    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:31:07.103145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:07.103112    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:31:07.387097 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:07.387010    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:07.563120 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:07.563085    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.991644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:11.991576    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" podUID="0cbe2544-9324-459c-b40f-5c6c95aff125" containerName="payload-processing" containerID="cri-o://d2db65c8efbdae32213da521f7c8a0102aa7ed90ff7b9651383d181e1dc4d89e" gracePeriod=25
Oct 30 22:31:12.104282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.104225    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" podUID="df2dbd35-2da5-40a3-9818-79c5428801f2" containerName="payload-pre-processing" containerID="cri-o://7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1" gracePeriod=25
Oct 30 22:31:12.157290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.157257    5510 generic.go:356] "Generic (PLEG): container finished" podID="0cbe2544-9324-459c-b40f-5c6c95aff125" containerID="d2db65c8efbdae32213da521f7c8a0102aa7ed90ff7b9651383d181e1dc4d89e" exitCode=0
Oct 30 22:31:12.157447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.157334    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" event={"ID":"0cbe2544-9324-459c-b40f-5c6c95aff125","Type":"ContainerDied","Data":"d2db65c8efbdae32213da521f7c8a0102aa7ed90ff7b9651383d181e1dc4d89e"}
Oct 30 22:31:12.242963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.242906    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:12.248057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.248038    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:31:12.352160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.352137    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:31:12.399975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.399941    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\") pod \"0cbe2544-9324-459c-b40f-5c6c95aff125\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") "
Oct 30 22:31:12.400133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.400055    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\" (UniqueName: \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\") pod \"0cbe2544-9324-459c-b40f-5c6c95aff125\" (UID: \"0cbe2544-9324-459c-b40f-5c6c95aff125\") "
Oct 30 22:31:12.400374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.400350    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume" pod "0cbe2544-9324-459c-b40f-5c6c95aff125" (UID: "0cbe2544-9324-459c-b40f-5c6c95aff125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:12.400464 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.400449    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cbe2544-9324-459c-b40f-5c6c95aff125-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:12.402042 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.402020    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b" pod "0cbe2544-9324-459c-b40f-5c6c95aff125" (UID: "0cbe2544-9324-459c-b40f-5c6c95aff125"). InnerVolumeSpecName "kube-api-access-qwg8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:12.501033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.500959    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\" (UniqueName: \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\") pod \"df2dbd35-2da5-40a3-9818-79c5428801f2\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") "
Oct 30 22:31:12.501168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.501085    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\") pod \"df2dbd35-2da5-40a3-9818-79c5428801f2\" (UID: \"df2dbd35-2da5-40a3-9818-79c5428801f2\") "
Oct 30 22:31:12.501265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.501253    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwg8b\" (UniqueName: \"kubernetes.io/projected/0cbe2544-9324-459c-b40f-5c6c95aff125-kube-api-access-qwg8b\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:12.501462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.501439    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume" pod "df2dbd35-2da5-40a3-9818-79c5428801f2" (UID: "df2dbd35-2da5-40a3-9818-79c5428801f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:12.503044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.503026    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84" pod "df2dbd35-2da5-40a3-9818-79c5428801f2" (UID: "df2dbd35-2da5-40a3-9818-79c5428801f2"). InnerVolumeSpecName "kube-api-access-fbt84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:12.602252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.602220    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df2dbd35-2da5-40a3-9818-79c5428801f2-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:12.602252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:12.602247    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt84\" (UniqueName: \"kubernetes.io/projected/df2dbd35-2da5-40a3-9818-79c5428801f2-kube-api-access-fbt84\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:13.164025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.163996    5510 generic.go:356] "Generic (PLEG): container finished" podID="df2dbd35-2da5-40a3-9818-79c5428801f2" containerID="7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1" exitCode=0
Oct 30 22:31:13.164417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.164051    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"
Oct 30 22:31:13.164417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.164070    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" event={"ID":"df2dbd35-2da5-40a3-9818-79c5428801f2","Type":"ContainerDied","Data":"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"}
Oct 30 22:31:13.164417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.164122    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj" event={"ID":"df2dbd35-2da5-40a3-9818-79c5428801f2","Type":"ContainerDied","Data":"8c52719790633730001e5adece6f318aa9cd85d38ba87a7f837a641f6816ff5c"}
Oct 30 22:31:13.164417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.164144    5510 scope.go:122] "RemoveContainer" containerID="7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"
Oct 30 22:31:13.165668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.165643    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf" event={"ID":"0cbe2544-9324-459c-b40f-5c6c95aff125","Type":"ContainerDied","Data":"6bacd684bed2f304e3e31fa6ee4d84a82e6d36fdabebab9d693055f5b02d2657"}
Oct 30 22:31:13.165668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.165667    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"
Oct 30 22:31:13.172872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.172844    5510 scope.go:122] "RemoveContainer" containerID="7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"
Oct 30 22:31:13.173153 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:13.173134    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1\": container with ID starting with 7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1 not found: ID does not exist" containerID="7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"
Oct 30 22:31:13.173218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.173160    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1"} err="failed to get container status \"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1\": rpc error: code = NotFound desc = could not find container \"7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1\": container with ID starting with 7bafb9bfe93d9b471cc6978d80476363b87db171ad05651e7c31db66da45eee1 not found: ID does not exist"
Oct 30 22:31:13.173218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.173176    5510 scope.go:122] "RemoveContainer" containerID="d2db65c8efbdae32213da521f7c8a0102aa7ed90ff7b9651383d181e1dc4d89e"
Oct 30 22:31:13.279114 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:13.279038    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:31:13.315136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.315106    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:31:13.318546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.318526    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-02bd3b-6cc9df6746-j2slf"]
Oct 30 22:31:13.419077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.419048    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:31:13.423732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:13.423708    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-02bd3b-967f6c68c-vtvdj"]
Oct 30 22:31:14.978808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:14.978762    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:15.123021 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:15.122986    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbe2544-9324-459c-b40f-5c6c95aff125" path="/var/lib/kubelet/pods/0cbe2544-9324-459c-b40f-5c6c95aff125/volumes"
Oct 30 22:31:15.123387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:15.123374    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2dbd35-2da5-40a3-9818-79c5428801f2" path="/var/lib/kubelet/pods/df2dbd35-2da5-40a3-9818-79c5428801f2/volumes"
Oct 30 22:31:15.126745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:15.126720    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:16.278533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.278500    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:31:16.279032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.278697    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" podUID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" containerName="maas-api" containerID="cri-o://5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e" gracePeriod=30
Oct 30 22:31:16.399911 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.399882    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:16.526827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.526798    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:31:16.645275 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.645204    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\") pod \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") "
Oct 30 22:31:16.645450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.645299    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\" (UniqueName: \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\") pod \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") "
Oct 30 22:31:16.645450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.645333    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") pod \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") "
Oct 30 22:31:16.645450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.645384    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\") pod \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\" (UID: \"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1\") "
Oct 30 22:31:16.645923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.645892    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca" pod "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" (UID: "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:16.647507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.647475    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs" pod "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" (UID: "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:16.647913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.647892    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt" pod "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" (UID: "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1"). InnerVolumeSpecName "kube-api-access-4lrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.651737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.651714    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls" pod "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" (UID: "187ccd21-bb05-49cd-9f6a-8f3b2795b1e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:16.746938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.746904    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.746938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.746931    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrkt\" (UniqueName: \"kubernetes.io/projected/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-kube-api-access-4lrkt\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.746938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.746943    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.747158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:16.746951    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:17.195447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.195409    5510 generic.go:356] "Generic (PLEG): container finished" podID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" containerID="5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e" exitCode=0
Oct 30 22:31:17.195818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.195795    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" event={"ID":"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1","Type":"ContainerDied","Data":"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"}
Oct 30 22:31:17.196029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.196012    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb" event={"ID":"187ccd21-bb05-49cd-9f6a-8f3b2795b1e1","Type":"ContainerDied","Data":"86d7d3cc0fb3b515031fa10775800c4d47a13cd6bde8b9950f1b015e99fe64d9"}
Oct 30 22:31:17.196160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.196145    5510 scope.go:122] "RemoveContainer" containerID="5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"
Oct 30 22:31:17.196615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.196587    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"
Oct 30 22:31:17.212168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.212141    5510 scope.go:122] "RemoveContainer" containerID="5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"
Oct 30 22:31:17.212499 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:17.212469    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e\": container with ID starting with 5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e not found: ID does not exist" containerID="5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"
Oct 30 22:31:17.212601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.212506    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e"} err="failed to get container status \"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e\": rpc error: code = NotFound desc = could not find container \"5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e\": container with ID starting with 5a41e7c07228e6714a53f9bfc1d3bd313b1bb902c9f86de830192ccc5cb5b10e not found: ID does not exist"
Oct 30 22:31:17.350926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.350889    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:31:17.396875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.396798    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-02bd3b-84ff47b8b5-59mrb"]
Oct 30 22:31:17.954528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:17.954483    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:18.157408 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:18.157367    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:18.306443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:18.306403    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:18.876429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:18.876401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:19.123896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:19.123867    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" path="/var/lib/kubelet/pods/187ccd21-bb05-49cd-9f6a-8f3b2795b1e1/volumes"
Oct 30 22:31:21.852325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:21.852240    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:31:23.170547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:23.170508    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.543606 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:24.543567    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.702039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:24.701986    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.842590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:24.842511    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:31:24.842745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:24.842714    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy" containerID="cri-o://8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a" gracePeriod=30
Oct 30 22:31:24.869917 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:24.869890    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.328543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:25.328507    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:28.196075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:28.196046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.086826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.086775    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:31:30.178827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178729    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.178827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178765    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.178827 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178814    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178842    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178888    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.178974    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179015    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179024    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179045    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179096    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\") pod \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\" (UID: \"9d015c53-755f-4af5-a5b0-cce8bf55ba00\") "
Oct 30 22:31:30.179115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179093    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179285    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179305    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179495    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179513    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179533    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179547    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.179572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.179561    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.181155 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.181134    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:30.181534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.181504    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "kube-api-access-5pj69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.181734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.181718    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.181923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.181901    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy" pod "9d015c53-755f-4af5-a5b0-cce8bf55ba00" (UID: "9d015c53-755f-4af5-a5b0-cce8bf55ba00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.263071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.263032    5510 generic.go:356] "Generic (PLEG): container finished" podID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerID="8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a" exitCode=0
Oct 30 22:31:30.263231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.263081    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"
Oct 30 22:31:30.263231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.263105    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerDied","Data":"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"}
Oct 30 22:31:30.263231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.263157    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv" event={"ID":"9d015c53-755f-4af5-a5b0-cce8bf55ba00","Type":"ContainerDied","Data":"8a3a51e6349a2af38b35fec131dffd4e6c4685d2d7f0e860408c09c9c5b67432"}
Oct 30 22:31:30.263231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.263180    5510 scope.go:122] "RemoveContainer" containerID="8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"
Oct 30 22:31:30.273104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.273086    5510 scope.go:122] "RemoveContainer" containerID="956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"
Oct 30 22:31:30.280391 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280370    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.280485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280394    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.280485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280406    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.280485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280416    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pj69\" (UniqueName: \"kubernetes.io/projected/9d015c53-755f-4af5-a5b0-cce8bf55ba00-kube-api-access-5pj69\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.280485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280424    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d015c53-755f-4af5-a5b0-cce8bf55ba00-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.280865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.280846    5510 scope.go:122] "RemoveContainer" containerID="8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"
Oct 30 22:31:30.281111 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:30.281088    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a\": container with ID starting with 8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a not found: ID does not exist" containerID="8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"
Oct 30 22:31:30.281192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.281122    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a"} err="failed to get container status \"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a\": rpc error: code = NotFound desc = could not find container \"8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a\": container with ID starting with 8adddda6be7704ca6f2784a080da7e2d34dbc69239cc8f0c08082f4ceb35286a not found: ID does not exist"
Oct 30 22:31:30.281192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.281145    5510 scope.go:122] "RemoveContainer" containerID="956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"
Oct 30 22:31:30.281385 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:30.281369    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e\": container with ID starting with 956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e not found: ID does not exist" containerID="956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"
Oct 30 22:31:30.281426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.281390    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e"} err="failed to get container status \"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e\": rpc error: code = NotFound desc = could not find container \"956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e\": container with ID starting with 956b3e62dd3fffe912d7b2afdb94664fa30cf039affc38f5b7f0ac4d0b23254e not found: ID does not exist"
Oct 30 22:31:30.289815 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.289596    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:31:30.291597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.291576    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 22:31:30.293098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.292767    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 22:31:30.293765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.293744    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 22:31:30.388040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.388010    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:30.736208 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.736178    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:31:30.742138 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:30.742110    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-02bd3b-openshift-default-5855c95d9-9vvxv"]
Oct 30 22:31:31.125850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:31.125762    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" path="/var/lib/kubelet/pods/9d015c53-755f-4af5-a5b0-cce8bf55ba00/volumes"
Oct 30 22:31:33.257561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:33.257529    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:33.415383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:33.415349    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:37.711988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:37.711955    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:37.855660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:37.855628    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:38.061618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:38.061512    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:42.342009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.341969    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"]
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342334    5510 cpu_manager.go:414] "containerMap: removing container" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342346    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342359    5510 cpu_manager.go:414] "containerMap: removing container" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342364    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342371    5510 cpu_manager.go:414] "containerMap: removing container" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342377    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342388    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df2dbd35-2da5-40a3-9818-79c5428801f2" containerName="payload-pre-processing"
Oct 30 22:31:42.342392 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342393    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2dbd35-2da5-40a3-9818-79c5428801f2" containerName="payload-pre-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342401    5510 cpu_manager.go:414] "containerMap: removing container" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342406    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342415    5510 cpu_manager.go:414] "containerMap: removing container" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342419    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342430    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0cbe2544-9324-459c-b40f-5c6c95aff125" containerName="payload-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342435    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbe2544-9324-459c-b40f-5c6c95aff125" containerName="payload-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342441    5510 cpu_manager.go:414] "containerMap: removing container" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342445    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342452    5510 cpu_manager.go:414] "containerMap: removing container" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342458    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342464    5510 cpu_manager.go:414] "containerMap: removing container" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342468    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342475    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d5e40124-3293-439d-89d0-082236c776d1" containerName="payload-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342480    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e40124-3293-439d-89d0-082236c776d1" containerName="payload-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342487    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342492    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a26591-e9d7-4d21-8f4f-431a79306b87" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342507    5510 cpu_manager.go:414] "containerMap: removing container" podUID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" containerName="payload-pre-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342512    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" containerName="payload-pre-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342521    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342525    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342531    5510 cpu_manager.go:414] "containerMap: removing container" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342535    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342542    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342546    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342552    5510 cpu_manager.go:414] "containerMap: removing container" podUID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342557    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342655    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfb9742-e8af-4b16-91c8-01e9b08c84b3" containerName="payload-pre-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342665    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21e5add-7f01-4b1c-a8f1-c3071bc83e2e" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342672    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342677    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342683    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342691    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="187ccd21-bb05-49cd-9f6a-8f3b2795b1e1" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342698    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="083c59b3-fcea-4225-b6a9-e46fdd1f8b67" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342703    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342708    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="544ea754-1458-4660-939e-4f974fc3cd17" containerName="istio-proxy"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342714    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342718    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2dbd35-2da5-40a3-9818-79c5428801f2" containerName="payload-pre-processing"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342723    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="02863ada-09c7-4996-9148-4844e22554f2" containerName="maas-api"
Oct 30 22:31:42.342670 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342729    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5290ba-2fc1-4fd3-bc6f-5d93e2f7c581" containerName="istio-proxy"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342754    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342762    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbe2544-9324-459c-b40f-5c6c95aff125" containerName="payload-processing"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342767    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342774    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a1e52b-8886-416a-ac49-e6a6f117559e" containerName="istio-proxy"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342795    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="69463924-211a-4117-bb6f-eec4aa88a760" containerName="istio-proxy"
Oct 30 22:31:42.344437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.342804    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e40124-3293-439d-89d0-082236c776d1" containerName="payload-processing"
Oct 30 22:31:42.346073 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.346049    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.402691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.402664    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"]
Oct 30 22:31:42.443240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.443213    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:42.443240 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.443216    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:42.443436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.443284    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:42.443436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.443379    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:31:42.495775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.495738    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj8x\" (UniqueName: \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.495959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.495804    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.495959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.495912    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.596623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.596535    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj8x\" (UniqueName: \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.596623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.596579    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.596623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.596599    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.597222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.597187    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.598978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.598957    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.651079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.651047    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj8x\" (UniqueName: \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\") pod \"maas-api-revoke-sub-ec15aa31-tf7w9\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:42.661861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:42.661834    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:43.032467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:43.032430    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"]
Oct 30 22:31:43.040178 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:31:43.040149    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b4ebdd_ae00_41ba_b4e0_0530b7c97c58.slice/crio-969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2 WatchSource:0}: Error finding container 969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2: Status 404 returned error can't find the container with id 969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2
Oct 30 22:31:43.286547 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:43.286474    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:31:43.323280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:43.323243    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9" event={"ID":"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58","Type":"ContainerStarted","Data":"969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2"}
Oct 30 22:31:44.329848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:44.329757    5510 generic.go:356] "Generic (PLEG): container finished" podID="d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" containerID="e38ac69ac06121537f662d4dec05f7ab72392ca5d9723c4f9af72cd4792dbb49" exitCode=0
Oct 30 22:31:44.329848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:44.329823    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9" event={"ID":"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58","Type":"ContainerDied","Data":"e38ac69ac06121537f662d4dec05f7ab72392ca5d9723c4f9af72cd4792dbb49"}
Oct 30 22:31:45.311502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.311460    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main" containerID="cri-o://28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790" gracePeriod=45
Oct 30 22:31:45.312100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.312067    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main" containerID="cri-o://9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517" gracePeriod=45
Oct 30 22:31:45.314286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.314255    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main" containerID="cri-o://2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74" gracePeriod=45
Oct 30 22:31:45.320250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.320221    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" containerID="cri-o://02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188" gracePeriod=45
Oct 30 22:31:45.324190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.324161    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": EOF"
Oct 30 22:31:45.570398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.570013    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:45.626438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.626413    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:31:45.637671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.637648    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:31:45.647290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.647269    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:31:45.651223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.651202    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:31:45.726637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726604    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\" (UniqueName: \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.726637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726638    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726670    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726698    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726730    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\" (UniqueName: \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726766    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726833    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\") pod \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") "
Oct 30 22:31:45.726890 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726864    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726907    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\" (UniqueName: \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726942    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.726991    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727001    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727005    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727030    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727085    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727102    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727142    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727177    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\") pod \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") "
Oct 30 22:31:45.727221 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727215    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.727858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727215    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.727858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727369    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.727858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727674    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca" pod "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" (UID: "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:45.728047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.727905    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.728124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.728101    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.729572 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729537    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca" pod "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" (UID: "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729573    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729579    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "kube-api-access-xbdzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729602    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\" (UniqueName: \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729646    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729680    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.729715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729714    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729749    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\") pod \"aa4864ae-619c-46c3-a3f5-ca5143683565\" (UID: \"aa4864ae-619c-46c3-a3f5-ca5143683565\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729817    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\") pod \"df3f0ff8-7975-458b-9af4-7e31beae3e52\" (UID: \"df3f0ff8-7975-458b-9af4-7e31beae3e52\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729851    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729900    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729925    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\" (UniqueName: \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\") pod \"2c12bd71-df32-420c-939a-31315e6b669d\" (UID: \"2c12bd71-df32-420c-939a-31315e6b669d\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730003    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\" (UniqueName: \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\") pod \"85b45b54-e0d9-494c-9e1c-93dc13f87804\" (UID: \"85b45b54-e0d9-494c-9e1c-93dc13f87804\") "
Oct 30 22:31:45.730089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730037    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\" (UniqueName: \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\") pod \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\" (UID: \"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58\") "
Oct 30 22:31:45.730451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729645    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.730451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.729968    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.730451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730219    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.730451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730425    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir" pod "df3f0ff8-7975-458b-9af4-7e31beae3e52" (UID: "df3f0ff8-7975-458b-9af4-7e31beae3e52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.730451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730445    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.730704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730530    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.730704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730611    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.730847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.730825    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.732018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.731988    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "kube-api-access-sclxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.732018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732010    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm" pod "aa4864ae-619c-46c3-a3f5-ca5143683565" (UID: "aa4864ae-619c-46c3-a3f5-ca5143683565"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.732185 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732055    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732189    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b45b54-e0d9-494c-9e1c-93dc13f87804-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732202    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732211    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732221    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732230    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbdzf\" (UniqueName: \"kubernetes.io/projected/df3f0ff8-7975-458b-9af4-7e31beae3e52-kube-api-access-xbdzf\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732238    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4864ae-619c-46c3-a3f5-ca5143683565-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732247    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c12bd71-df32-420c-939a-31315e6b669d-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732255    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732263    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732270    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732277    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732285    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732294    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclxw\" (UniqueName: \"kubernetes.io/projected/aa4864ae-619c-46c3-a3f5-ca5143683565-kube-api-access-sclxw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732302    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732310    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732317    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3f0ff8-7975-458b-9af4-7e31beae3e52-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732326    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732333    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732340    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3f0ff8-7975-458b-9af4-7e31beae3e52-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732347    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732355    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732364    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4864ae-619c-46c3-a3f5-ca5143683565-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.732730 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732527    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.733296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732808    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs" pod "85b45b54-e0d9-494c-9e1c-93dc13f87804" (UID: "85b45b54-e0d9-494c-9e1c-93dc13f87804"). InnerVolumeSpecName "kube-api-access-sqjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.733296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.732874    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:45.733296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.733207    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd" pod "2c12bd71-df32-420c-939a-31315e6b669d" (UID: "2c12bd71-df32-420c-939a-31315e6b669d"). InnerVolumeSpecName "kube-api-access-8stjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.733425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.733325    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x" pod "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" (UID: "d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58"). InnerVolumeSpecName "kube-api-access-mlj8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:45.833646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.833573    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b45b54-e0d9-494c-9e1c-93dc13f87804-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.833646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.833598    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c12bd71-df32-420c-939a-31315e6b669d-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.833646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.833610    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8stjd\" (UniqueName: \"kubernetes.io/projected/2c12bd71-df32-420c-939a-31315e6b669d-kube-api-access-8stjd\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.833646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.833620    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjfs\" (UniqueName: \"kubernetes.io/projected/85b45b54-e0d9-494c-9e1c-93dc13f87804-kube-api-access-sqjfs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:45.833646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:45.833631    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlj8x\" (UniqueName: \"kubernetes.io/projected/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58-kube-api-access-mlj8x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:46.341429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.341396    5510 generic.go:356] "Generic (PLEG): container finished" podID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerID="2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74" exitCode=0
Oct 30 22:31:46.341612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.341450    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"
Oct 30 22:31:46.341612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.341487    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" event={"ID":"aa4864ae-619c-46c3-a3f5-ca5143683565","Type":"ContainerDied","Data":"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"}
Oct 30 22:31:46.341612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.341536    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8" event={"ID":"aa4864ae-619c-46c3-a3f5-ca5143683565","Type":"ContainerDied","Data":"35082210807b81670e9d48e749a41de1a3778e08d95ed656c01a0c4269e05402"}
Oct 30 22:31:46.341612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.341554    5510 scope.go:122] "RemoveContainer" containerID="2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"
Oct 30 22:31:46.342843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.342748    5510 generic.go:356] "Generic (PLEG): container finished" podID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerID="9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517" exitCode=0
Oct 30 22:31:46.342843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.342811    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" event={"ID":"df3f0ff8-7975-458b-9af4-7e31beae3e52","Type":"ContainerDied","Data":"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"}
Oct 30 22:31:46.342843 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.342829    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"
Oct 30 22:31:46.342999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.342852    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl" event={"ID":"df3f0ff8-7975-458b-9af4-7e31beae3e52","Type":"ContainerDied","Data":"bc2cdc140c1e15a4b737e5826adf1ff8590a14905dd817f31c24ca447754ec3b"}
Oct 30 22:31:46.346015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.345990    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9" event={"ID":"d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58","Type":"ContainerDied","Data":"969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2"}
Oct 30 22:31:46.346118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.346014    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"
Oct 30 22:31:46.346118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.346038    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969e227beb3c25d444b2cbdba3f8113b5bcb5f3d2a3353392461b6ee7d477aa2"
Oct 30 22:31:46.347594 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.347572    5510 generic.go:356] "Generic (PLEG): container finished" podID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerID="02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188" exitCode=0
Oct 30 22:31:46.347706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.347633    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"
Oct 30 22:31:46.347706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.347671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" event={"ID":"85b45b54-e0d9-494c-9e1c-93dc13f87804","Type":"ContainerDied","Data":"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"}
Oct 30 22:31:46.347706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.347702    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd" event={"ID":"85b45b54-e0d9-494c-9e1c-93dc13f87804","Type":"ContainerDied","Data":"486aa00e08cf0a0ae787df69976c5dd87885244bacfd5a3469dacc290bc5131e"}
Oct 30 22:31:46.349160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.349137    5510 generic.go:356] "Generic (PLEG): container finished" podID="2c12bd71-df32-420c-939a-31315e6b669d" containerID="28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790" exitCode=0
Oct 30 22:31:46.349267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.349189    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" event={"ID":"2c12bd71-df32-420c-939a-31315e6b669d","Type":"ContainerDied","Data":"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"}
Oct 30 22:31:46.349267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.349218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q" event={"ID":"2c12bd71-df32-420c-939a-31315e6b669d","Type":"ContainerDied","Data":"b3a2ddeda10f265ba9b7bb14c0e246117104d4689c21e9b082b0812ff6e2213b"}
Oct 30 22:31:46.349267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.349193    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"
Oct 30 22:31:46.351258 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.351240    5510 scope.go:122] "RemoveContainer" containerID="2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"
Oct 30 22:31:46.351534 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:46.351505    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74\": container with ID starting with 2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74 not found: ID does not exist" containerID="2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"
Oct 30 22:31:46.351641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.351544    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74"} err="failed to get container status \"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74\": rpc error: code = NotFound desc = could not find container \"2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74\": container with ID starting with 2e275ba6c1a41fc86751e473f06dab200d9021a7932de1320416a0e53e88ee74 not found: ID does not exist"
Oct 30 22:31:46.351641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.351568    5510 scope.go:122] "RemoveContainer" containerID="9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"
Oct 30 22:31:46.361168 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.361152    5510 scope.go:122] "RemoveContainer" containerID="9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"
Oct 30 22:31:46.361393 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:46.361374    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517\": container with ID starting with 9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517 not found: ID does not exist" containerID="9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"
Oct 30 22:31:46.361493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.361402    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517"} err="failed to get container status \"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517\": rpc error: code = NotFound desc = could not find container \"9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517\": container with ID starting with 9fb5566e7669161ef17d27732d90ca060cabaaa0bc2f532668310b4b57a80517 not found: ID does not exist"
Oct 30 22:31:46.361493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.361422    5510 scope.go:122] "RemoveContainer" containerID="02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"
Oct 30 22:31:46.369493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.369476    5510 scope.go:122] "RemoveContainer" containerID="02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"
Oct 30 22:31:46.369717 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:46.369698    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188\": container with ID starting with 02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188 not found: ID does not exist" containerID="02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"
Oct 30 22:31:46.369789 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.369723    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188"} err="failed to get container status \"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188\": rpc error: code = NotFound desc = could not find container \"02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188\": container with ID starting with 02059d7ac00715fb6c6e5183050125285966c73b92588da37c5d72c04d299188 not found: ID does not exist"
Oct 30 22:31:46.369789 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.369741    5510 scope.go:122] "RemoveContainer" containerID="28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"
Oct 30 22:31:46.377004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.376987    5510 scope.go:122] "RemoveContainer" containerID="28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"
Oct 30 22:31:46.377257 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:31:46.377239    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790\": container with ID starting with 28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790 not found: ID does not exist" containerID="28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"
Oct 30 22:31:46.377307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.377264    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790"} err="failed to get container status \"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790\": rpc error: code = NotFound desc = could not find container \"28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790\": container with ID starting with 28819291d1cbe0e594f97be84b22cc4c2a5c99f908bd6b98ae8ff995ed325790 not found: ID does not exist"
Oct 30 22:31:46.491424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.491393    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 22:31:46.537351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.537318    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/simulator-w5-5402dd-kserve-76d6d56c45-zsrfd"]
Oct 30 22:31:46.686520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.686436    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 22:31:46.694629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.694603    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-w5-5402dd-kserve-84f596b5bc-6z98q"]
Oct 30 22:31:46.795237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.795207    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 22:31:46.798973 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.798947    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/premium-w5-5402dd-kserve-79f596689b-q9xcl"]
Oct 30 22:31:46.905985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.905942    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:31:46.910448 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:46.910423    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5402dd/distinct-2-w5-5402dd-kserve-77566b86dd-55xd8"]
Oct 30 22:31:47.124622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:47.124589    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c12bd71-df32-420c-939a-31315e6b669d" path="/var/lib/kubelet/pods/2c12bd71-df32-420c-939a-31315e6b669d/volumes"
Oct 30 22:31:47.125053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:47.125032    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" path="/var/lib/kubelet/pods/85b45b54-e0d9-494c-9e1c-93dc13f87804/volumes"
Oct 30 22:31:47.125459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:47.125444    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" path="/var/lib/kubelet/pods/aa4864ae-619c-46c3-a3f5-ca5143683565/volumes"
Oct 30 22:31:47.126318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:47.126297    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" path="/var/lib/kubelet/pods/df3f0ff8-7975-458b-9af4-7e31beae3e52/volumes"
Oct 30 22:31:57.740956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:57.740926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.171642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:59.171607    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:31:59.249328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:59.249277    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.551409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:31:59.551377    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.200330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:00.200298    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:06.698481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:06.698441    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:06.861766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:06.861733    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.014003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:07.013942    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.178295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:07.178262    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:08.523309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:08.523277    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:13.001189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.001152    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:32:13.279075 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:32:13.279003    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:32:13.989562 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.989516    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:32:13.989954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.989934    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:32:13.989954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.989955    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d015c53-755f-4af5-a5b0-cce8bf55ba00" containerName="istio-proxy"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.989978    5510 cpu_manager.go:414] "containerMap: removing container" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.989984    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335da42c-dbf7-4e8e-bef8-21460bdcddf7" containerName="cleanup"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990002    5510 cpu_manager.go:414] "containerMap: removing container" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990009    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990016    5510 cpu_manager.go:414] "containerMap: removing container" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990021    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990037    5510 cpu_manager.go:414] "containerMap: removing container" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990044    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990073    5510 cpu_manager.go:414] "containerMap: removing container" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990079    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990088    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" containerName="revoke-keys"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990092    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" containerName="revoke-keys"
Oct 30 22:32:13.990150 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990151    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3f0ff8-7975-458b-9af4-7e31beae3e52" containerName="main"
Oct 30 22:32:13.990756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990161    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" containerName="revoke-keys"
Oct 30 22:32:13.990756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990176    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b45b54-e0d9-494c-9e1c-93dc13f87804" containerName="main"
Oct 30 22:32:13.990756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990185    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4864ae-619c-46c3-a3f5-ca5143683565" containerName="main"
Oct 30 22:32:13.990756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.990196    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c12bd71-df32-420c-939a-31315e6b669d" containerName="main"
Oct 30 22:32:13.993282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:13.993265    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.077534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.077499    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hflx\" (UniqueName: \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.078016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.077551    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.078016 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.077707    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.087768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.087742    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 22:32:14.087768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.087754    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 22:32:14.087946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.087766    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 22:32:14.087946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.087748    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lrchh\""
Oct 30 22:32:14.089579 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.089546    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:32:14.178664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.178615    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hflx\" (UniqueName: \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.178845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.178700    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.178845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.178802    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.181157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.181133    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.181286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.181174    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.234003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.233974    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hflx\" (UniqueName: \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\") pod \"maas-controller-5844bc79b5-nl824\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") " pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.270847 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.270793    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:14.307389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.307367    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:14.676954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.676883    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:32:14.715529 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:32:14.715501    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffe0058_b8e1_4b74_97a6_5e795b628ef5.slice/crio-4ea9b8da94930c37108bda7203fb42fd10d4a1bc1a01f431e72c6f0b7e7933f5 WatchSource:0}: Error finding container 4ea9b8da94930c37108bda7203fb42fd10d4a1bc1a01f431e72c6f0b7e7933f5: Status 404 returned error can't find the container with id 4ea9b8da94930c37108bda7203fb42fd10d4a1bc1a01f431e72c6f0b7e7933f5
Oct 30 22:32:14.716756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:14.716739    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:32:15.479583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:15.479549    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerStarted","Data":"4ea9b8da94930c37108bda7203fb42fd10d4a1bc1a01f431e72c6f0b7e7933f5"}
Oct 30 22:32:17.491299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:17.491247    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerStarted","Data":"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"}
Oct 30 22:32:17.491709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:17.491318    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:17.601770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:17.601707    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5844bc79b5-nl824" podStartSLOduration=2.499484737 podStartE2EDuration="4.601690299s" totalImagesPullingTime="2.102205562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:17.600642404 +0000 UTC m=+2946.584549284" watchObservedRunningTime="2026-10-30 22:32:17.601690299 +0000 UTC m=+2946.585597178"
Oct 30 22:32:27.537437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:27.537399    5510 generic.go:356] "Generic (PLEG): container finished" podID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerID="c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8" exitCode=1
Oct 30 22:32:27.537887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:27.537455    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerDied","Data":"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"}
Oct 30 22:32:27.537887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:27.537822    5510 scope.go:122] "RemoveContainer" containerID="c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"
Oct 30 22:32:28.543140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:28.543107    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerStarted","Data":"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"}
Oct 30 22:32:28.543626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:28.543405    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:33.727462 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:33.727430    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:36.163337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.163301    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"]
Oct 30 22:32:36.166861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.166845    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.209485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.209451    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"]
Oct 30 22:32:36.213178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.213160    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.224022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.223995    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d6th4\""
Oct 30 22:32:36.262549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.262522    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"]
Oct 30 22:32:36.272330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272303    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.272470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272415    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.272550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272516    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.272619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272580    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a829d55-09d6-4110-ab64-7b84c761319f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.273018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272846    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.273018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272893    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.273018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.272948    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfw2\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-kube-api-access-7bfw2\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.273256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.273088    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.273256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.273154    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a829d55-09d6-4110-ab64-7b84c761319f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.309447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.309415    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"]
Oct 30 22:32:36.374171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374135    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374180    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a829d55-09d6-4110-ab64-7b84c761319f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374204    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374342    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374378    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374401    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfw2\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-kube-api-access-7bfw2\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374421    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374438    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374465    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374499    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a829d55-09d6-4110-ab64-7b84c761319f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374585    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gmzk\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-kube-api-access-4gmzk\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374631    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c45657-26b2-4455-8d74-5031505f8482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374695    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c45657-26b2-4455-8d74-5031505f8482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374757    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374828    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374843    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.374902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374890    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.375141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374899    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.375141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374927    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.375141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.374986    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.375141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.375001    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.375287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.375263    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a829d55-09d6-4110-ab64-7b84c761319f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.375329 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.375290    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.376801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.376759    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a829d55-09d6-4110-ab64-7b84c761319f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.376914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.376875    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a829d55-09d6-4110-ab64-7b84c761319f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.428493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.428427    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.428615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.428555    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfw2\" (UniqueName: \"kubernetes.io/projected/3a829d55-09d6-4110-ab64-7b84c761319f-kube-api-access-7bfw2\") pod \"maas-default-gateway-openshift-default-694db55c87-8f4kg\" (UID: \"3a829d55-09d6-4110-ab64-7b84c761319f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.476153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476126    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476165    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476189    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476222    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476237    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476272    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gmzk\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-kube-api-access-4gmzk\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476289    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c45657-26b2-4455-8d74-5031505f8482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476308    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c45657-26b2-4455-8d74-5031505f8482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476331    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476519    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476563    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476777 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476647    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.476777 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.476657    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.477109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.477088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c45657-26b2-4455-8d74-5031505f8482-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.478452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.478430    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c45657-26b2-4455-8d74-5031505f8482-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.478566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.478546    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c45657-26b2-4455-8d74-5031505f8482-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.480528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.480514    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:36.532554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.532523    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gmzk\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-kube-api-access-4gmzk\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.532976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.532949    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c45657-26b2-4455-8d74-5031505f8482-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-525t7\" (UID: \"a5c45657-26b2-4455-8d74-5031505f8482\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.826549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.826501    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:36.900928 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.900900    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"]
Oct 30 22:32:36.909227 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:32:36.909201    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a829d55_09d6_4110_ab64_7b84c761319f.slice/crio-f8a3807a28380e424a99b6f43a40aec21606ac18ab023dbde7c6832415f2365a WatchSource:0}: Error finding container f8a3807a28380e424a99b6f43a40aec21606ac18ab023dbde7c6832415f2365a: Status 404 returned error can't find the container with id f8a3807a28380e424a99b6f43a40aec21606ac18ab023dbde7c6832415f2365a
Oct 30 22:32:36.911480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.911434    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:36.911581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.911529    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:36.911581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:36.911570    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:37.195687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.195601    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"]
Oct 30 22:32:37.203726 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:32:37.203695    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c45657_26b2_4455_8d74_5031505f8482.slice/crio-3fc8da1009ae47a1e23edd74a7b8f28bbd8f23e9a6f778bf77865470b348dfae WatchSource:0}: Error finding container 3fc8da1009ae47a1e23edd74a7b8f28bbd8f23e9a6f778bf77865470b348dfae: Status 404 returned error can't find the container with id 3fc8da1009ae47a1e23edd74a7b8f28bbd8f23e9a6f778bf77865470b348dfae
Oct 30 22:32:37.206565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.206533    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:37.206697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.206616    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:37.206697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.206661    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:37.585976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.585919    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" event={"ID":"a5c45657-26b2-4455-8d74-5031505f8482","Type":"ContainerStarted","Data":"b0aff117907dbead2f4a519b5d3de49054cb3b572ee55e3fd092f3f3953b2826"}
Oct 30 22:32:37.586196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.585999    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" event={"ID":"a5c45657-26b2-4455-8d74-5031505f8482","Type":"ContainerStarted","Data":"3fc8da1009ae47a1e23edd74a7b8f28bbd8f23e9a6f778bf77865470b348dfae"}
Oct 30 22:32:37.587684 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.587650    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" event={"ID":"3a829d55-09d6-4110-ab64-7b84c761319f","Type":"ContainerStarted","Data":"9188aaa7ba13d623fc7e26cae21ada50c886f0b59e385220b50c29fa6556a1f4"}
Oct 30 22:32:37.587845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.587700    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" event={"ID":"3a829d55-09d6-4110-ab64-7b84c761319f","Type":"ContainerStarted","Data":"f8a3807a28380e424a99b6f43a40aec21606ac18ab023dbde7c6832415f2365a"}
Oct 30 22:32:37.701164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.701114    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podStartSLOduration=1.701099946 podStartE2EDuration="1.701099946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:37.699776794 +0000 UTC m=+2966.683683669" watchObservedRunningTime="2026-10-30 22:32:37.701099946 +0000 UTC m=+2966.685006822"
Oct 30 22:32:37.816722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.816648    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podStartSLOduration=1.816631847 podStartE2EDuration="1.816631847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:37.815253445 +0000 UTC m=+2966.799160322" watchObservedRunningTime="2026-10-30 22:32:37.816631847 +0000 UTC m=+2966.800538717"
Oct 30 22:32:37.826631 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:37.826603    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:38.483803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:38.483496    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:38.483803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:38.483636    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:38.483803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:38.483696    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:38.829251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:38.829166    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:38.829251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:38.829226    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:39.483804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:39.483182    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:39.483804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:39.483245    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:39.549899 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:39.549866    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:39.827411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:39.827203    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:39.827411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:39.827271    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:40.481805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:40.481219    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:40.481805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:40.481282    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:40.827950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:40.827856    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:40.827950 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:40.827909    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:41.481805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:41.481166    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:41.481805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:41.481231    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:41.827419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:41.827331    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:41.827419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:41.827386    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:42.484256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:42.484212    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:42.484922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:42.484884    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:42.827201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:42.827114    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:42.827201 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:42.827167    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:43.281727 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:32:43.281683    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:32:43.482007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:43.481962    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:43.482191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:43.482037    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:43.827137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:43.827102    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:32:43.827528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:43.827160    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Oct 30 22:32:44.481698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:44.481659    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 22:32:44.481914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:44.481727    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 22:32:45.486062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.486017    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:45.486546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.486485    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:45.487378 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.487355    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg"
Oct 30 22:32:45.564433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.564393    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:45.565059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.565017    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:45.565675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:45.565645    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7"
Oct 30 22:32:51.295880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.295771    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"]
Oct 30 22:32:51.301896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.301875    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.324795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.324754    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.324944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.324873    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.324937    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.324985    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.325017    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngzk\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-kube-api-access-vngzk\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.325064    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.325088    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.325145    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.325225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.325178    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.397749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.397714    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"]
Oct 30 22:32:51.426674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426638    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.426879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426695    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.426879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426751    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.426879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426824    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.426879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426860    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngzk\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-kube-api-access-vngzk\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426907    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426932    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.426995    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427137 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427029    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427228    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427249    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427486    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427558    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.427764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.427742    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.429435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.429409    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.430086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.430064    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.479991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.479959    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.480122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.479967    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngzk\" (UniqueName: \"kubernetes.io/projected/0f7f5f7a-9377-4e4c-9399-492dedf7a100-kube-api-access-vngzk\") pod \"maas-default-gateway-openshift-default-694db55c87-ks5ks\" (UID: \"0f7f5f7a-9377-4e4c-9399-492dedf7a100\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:51.617735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:51.617643    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:52.045455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.045420    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"]
Oct 30 22:32:52.113801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.107015    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:52.113801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.108925    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:52.113801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.108986    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:32:52.676002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.675964    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" event={"ID":"0f7f5f7a-9377-4e4c-9399-492dedf7a100","Type":"ContainerStarted","Data":"ae5816533d635278d480ad4d1e9a9ca8c0d47c8aeb64c0a0df93b30d32e0d771"}
Oct 30 22:32:52.677452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.676565    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" event={"ID":"0f7f5f7a-9377-4e4c-9399-492dedf7a100","Type":"ContainerStarted","Data":"fda582c1041564e9d60e37bdf834d32a8c88d55384a9194f9178f85646e0ceb8"}
Oct 30 22:32:52.793175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:52.792813    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podStartSLOduration=1.7927740330000002 podStartE2EDuration="1.792774033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:52.790335285 +0000 UTC m=+2981.774242162" watchObservedRunningTime="2026-10-30 22:32:52.792774033 +0000 UTC m=+2981.776680911"
Oct 30 22:32:53.618111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:53.618062    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:32:53.619660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:53.619631    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:53.619821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:53.619683    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:54.618587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:54.618547    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:54.619077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:54.618608    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:55.618539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:55.618497    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:55.618722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:55.618555    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:56.618303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:56.618263    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:56.618491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:56.618321    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:57.618600 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:57.618561    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:57.619048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:57.618621    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:58.618232 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:58.618194    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:58.618445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:58.618248    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:58.730755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:58.730727    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:58.889217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:58.889133    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:32:58.889402 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:58.889368    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5844bc79b5-nl824" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager" containerID="cri-o://5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce" gracePeriod=30
Oct 30 22:32:59.547959 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.547913    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-5844bc79b5-nl824" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.213:8081/readyz\": dial tcp 10.133.0.213:8081: connect: connection refused"
Oct 30 22:32:59.618793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.618753    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 22:32:59.618926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.618852    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 22:32:59.632314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.632290    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:59.711910 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.711881    5510 generic.go:356] "Generic (PLEG): container finished" podID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerID="5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce" exitCode=0
Oct 30 22:32:59.712091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.711934    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-nl824"
Oct 30 22:32:59.712091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.711955    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerDied","Data":"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"}
Oct 30 22:32:59.712091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.711998    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-nl824" event={"ID":"cffe0058-b8e1-4b74-97a6-5e795b628ef5","Type":"ContainerDied","Data":"4ea9b8da94930c37108bda7203fb42fd10d4a1bc1a01f431e72c6f0b7e7933f5"}
Oct 30 22:32:59.712091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.712019    5510 scope.go:122] "RemoveContainer" containerID="5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"
Oct 30 22:32:59.720508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.720489    5510 scope.go:122] "RemoveContainer" containerID="c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"
Oct 30 22:32:59.728974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.728949    5510 scope.go:122] "RemoveContainer" containerID="5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"
Oct 30 22:32:59.729211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729190    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\") pod \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") "
Oct 30 22:32:59.729304 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:32:59.729226    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce\": container with ID starting with 5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce not found: ID does not exist" containerID="5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"
Oct 30 22:32:59.729304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729246    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\" (UniqueName: \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\") pod \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") "
Oct 30 22:32:59.729304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729267    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce"} err="failed to get container status \"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce\": rpc error: code = NotFound desc = could not find container \"5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce\": container with ID starting with 5e820d5ac8bb4ad94d8cf4c4d42517119319fb6f68616a08a4972876612ee5ce not found: ID does not exist"
Oct 30 22:32:59.729304 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729292    5510 scope.go:122] "RemoveContainer" containerID="c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"
Oct 30 22:32:59.729527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729326    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\") pod \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\" (UID: \"cffe0058-b8e1-4b74-97a6-5e795b628ef5\") "
Oct 30 22:32:59.729617 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:32:59.729592    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8\": container with ID starting with c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8 not found: ID does not exist" containerID="c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"
Oct 30 22:32:59.729687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.729629    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8"} err="failed to get container status \"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8\": rpc error: code = NotFound desc = could not find container \"c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8\": container with ID starting with c774cf33fffdefc354415ea165f31f0dbec96e7ad7f995fbac2e85fa38f54ee8 not found: ID does not exist"
Oct 30 22:32:59.731204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.731176    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx" pod "cffe0058-b8e1-4b74-97a6-5e795b628ef5" (UID: "cffe0058-b8e1-4b74-97a6-5e795b628ef5"). InnerVolumeSpecName "kube-api-access-4hflx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:59.731700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.731674    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert" pod "cffe0058-b8e1-4b74-97a6-5e795b628ef5" (UID: "cffe0058-b8e1-4b74-97a6-5e795b628ef5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:59.731877 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.731684    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs" pod "cffe0058-b8e1-4b74-97a6-5e795b628ef5" (UID: "cffe0058-b8e1-4b74-97a6-5e795b628ef5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:59.831268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.831180    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hflx\" (UniqueName: \"kubernetes.io/projected/cffe0058-b8e1-4b74-97a6-5e795b628ef5-kube-api-access-4hflx\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:59.831268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.831219    5510 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-webhook-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:59.831268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:32:59.831236    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cffe0058-b8e1-4b74-97a6-5e795b628ef5-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.125098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:00.125020    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:33:00.127271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:00.127245    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-nl824"]
Oct 30 22:33:00.622468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:00.622438    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:33:00.622745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:00.622724    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:33:00.623520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:00.623502    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks"
Oct 30 22:33:01.124000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.123968    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" path="/var/lib/kubelet/pods/cffe0058-b8e1-4b74-97a6-5e795b628ef5/volumes"
Oct 30 22:33:01.434259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.434179    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:01.556252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556216    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5844bc79b5-hj279"]
Oct 30 22:33:01.556609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556597    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:33:01.556653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556610    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:33:01.556653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556627    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:33:01.556653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556632    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:33:01.556753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.556705    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:33:01.561415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.561395    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.610041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.610007    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 22:33:01.610300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.610206    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lrchh\""
Oct 30 22:33:01.649426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.649386    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-metrics-certs\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.649609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.649441    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskll\" (UniqueName: \"kubernetes.io/projected/f10c1f15-c182-4754-a084-cab33f005873-kube-api-access-nskll\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.649609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.649517    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-webhook-cert\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.655978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.655950    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 22:33:01.655978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.655958    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 22:33:01.657484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.657461    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-hj279"]
Oct 30 22:33:01.750299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.750262    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskll\" (UniqueName: \"kubernetes.io/projected/f10c1f15-c182-4754-a084-cab33f005873-kube-api-access-nskll\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.750528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.750327    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-webhook-cert\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.750528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.750470    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-metrics-certs\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.752850 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.752822    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-metrics-certs\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.752968 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.752944    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f10c1f15-c182-4754-a084-cab33f005873-webhook-cert\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.804469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.804435    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskll\" (UniqueName: \"kubernetes.io/projected/f10c1f15-c182-4754-a084-cab33f005873-kube-api-access-nskll\") pod \"maas-controller-5844bc79b5-hj279\" (UID: \"f10c1f15-c182-4754-a084-cab33f005873\") " pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:01.874592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:01.874551    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:02.242545 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:02.242506    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5844bc79b5-hj279"]
Oct 30 22:33:02.280855 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:33:02.280820    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10c1f15_c182_4754_a084_cab33f005873.slice/crio-7bc1aae5e4ae0f4e92fd63a2191d8edad0ad2b738ddcdd53bfbc735b227cc8f4 WatchSource:0}: Error finding container 7bc1aae5e4ae0f4e92fd63a2191d8edad0ad2b738ddcdd53bfbc735b227cc8f4: Status 404 returned error can't find the container with id 7bc1aae5e4ae0f4e92fd63a2191d8edad0ad2b738ddcdd53bfbc735b227cc8f4
Oct 30 22:33:02.727702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:02.727667    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-hj279" event={"ID":"f10c1f15-c182-4754-a084-cab33f005873","Type":"ContainerStarted","Data":"7bc1aae5e4ae0f4e92fd63a2191d8edad0ad2b738ddcdd53bfbc735b227cc8f4"}
Oct 30 22:33:03.733972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:03.733931    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5844bc79b5-hj279" event={"ID":"f10c1f15-c182-4754-a084-cab33f005873","Type":"ContainerStarted","Data":"0cb4133adb28ff2551aadcaf7a9ec8e87b2c6e750cd12c96cd7d4058ca43f912"}
Oct 30 22:33:03.734399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:03.734016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:03.844308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:03.844244    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5844bc79b5-hj279" podStartSLOduration=1.949394214 podStartE2EDuration="2.844231434s" totalImagesPullingTime="894.83722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:03.842362829 +0000 UTC m=+2992.826269705" watchObservedRunningTime="2026-10-30 22:33:03.844231434 +0000 UTC m=+2992.828138360"
Oct 30 22:33:11.917758 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:11.917728    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:33:12.361898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:12.361868    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:33:13.278702 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:33:13.278671    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:33:14.741047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:14.741016    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5844bc79b5-hj279"
Oct 30 22:33:34.108043 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:34.108011    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:43.278981 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:33:43.278943    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:33:46.568677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:33:46.568639    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:34:07.945866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:07.945830    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:34:07.984610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:07.984578    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:13.291705 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:34:13.291649    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:34:18.857503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:18.857464    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:34:21.338523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:21.338449    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:26.327656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.327622    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"]
Oct 30 22:34:26.328302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.328121    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffe0058-b8e1-4b74-97a6-5e795b628ef5" containerName="manager"
Oct 30 22:34:26.330400 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.330377    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.379081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.379052    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:26.379081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.379052    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:26.425081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.425053    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:34:26.425271 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.425141    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:26.426474 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.426452    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"]
Oct 30 22:34:26.428417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.428400    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl45n\" (UniqueName: \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.428480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.428431    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.428480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.428460    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.529944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.529906    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl45n\" (UniqueName: \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.530151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.529952    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.530151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.529997    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.530660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.530635    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.532340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.532319    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.583831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.583761    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl45n\" (UniqueName: \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\") pod \"maas-api-revoke-sub-2e1f5b5f-cvhgd\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:26.642695 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:26.642658    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:27.008470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:27.008433    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"]
Oct 30 22:34:27.016694 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:34:27.016664    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a8e77e_9608_41be_9945_2758e26ff4ca.slice/crio-49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2 WatchSource:0}: Error finding container 49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2: Status 404 returned error can't find the container with id 49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2
Oct 30 22:34:27.158929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:27.158897    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd" event={"ID":"b2a8e77e-9608-41be-9945-2758e26ff4ca","Type":"ContainerStarted","Data":"49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2"}
Oct 30 22:34:28.165964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:28.165884    5510 generic.go:356] "Generic (PLEG): container finished" podID="b2a8e77e-9608-41be-9945-2758e26ff4ca" containerID="80b586175406b74f434e8fe6d1902bd21b325257aeb8ebb79f3b56e6f9914de3" exitCode=0
Oct 30 22:34:28.165964 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:28.165942    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd" event={"ID":"b2a8e77e-9608-41be-9945-2758e26ff4ca","Type":"ContainerDied","Data":"80b586175406b74f434e8fe6d1902bd21b325257aeb8ebb79f3b56e6f9914de3"}
Oct 30 22:34:29.298479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.298453    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:29.355117 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.355087    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\" (UniqueName: \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\") pod \"b2a8e77e-9608-41be-9945-2758e26ff4ca\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") "
Oct 30 22:34:29.355260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.355173    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\") pod \"b2a8e77e-9608-41be-9945-2758e26ff4ca\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") "
Oct 30 22:34:29.355260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.355245    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\") pod \"b2a8e77e-9608-41be-9945-2758e26ff4ca\" (UID: \"b2a8e77e-9608-41be-9945-2758e26ff4ca\") "
Oct 30 22:34:29.355589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.355568    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca" pod "b2a8e77e-9608-41be-9945-2758e26ff4ca" (UID: "b2a8e77e-9608-41be-9945-2758e26ff4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:29.357206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.357186    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca" pod "b2a8e77e-9608-41be-9945-2758e26ff4ca" (UID: "b2a8e77e-9608-41be-9945-2758e26ff4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:29.357273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.357211    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n" pod "b2a8e77e-9608-41be-9945-2758e26ff4ca" (UID: "b2a8e77e-9608-41be-9945-2758e26ff4ca"). InnerVolumeSpecName "kube-api-access-gl45n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:29.456157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.456091    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl45n\" (UniqueName: \"kubernetes.io/projected/b2a8e77e-9608-41be-9945-2758e26ff4ca-kube-api-access-gl45n\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:29.456157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.456119    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:29.456157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.456128    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a8e77e-9608-41be-9945-2758e26ff4ca-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:29.640560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:29.640529    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:30.176368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:30.176336    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"
Oct 30 22:34:30.176368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:30.176359    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd" event={"ID":"b2a8e77e-9608-41be-9945-2758e26ff4ca","Type":"ContainerDied","Data":"49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2"}
Oct 30 22:34:30.176671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:30.176402    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49c50357f48b8e5ddae3ef5e4b97ac7e158f5a7747dc353152eb2f31067fefb2"
Oct 30 22:34:30.332065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:30.332030    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:32.931551 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:32.931516    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:33.090487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:33.090458    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:34.390571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:34.390540    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:38.847024 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.846989    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"]
Oct 30 22:34:38.847432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.847399    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b2a8e77e-9608-41be-9945-2758e26ff4ca" containerName="revoke-keys"
Oct 30 22:34:38.847432 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.847410    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a8e77e-9608-41be-9945-2758e26ff4ca" containerName="revoke-keys"
Oct 30 22:34:38.847509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.847493    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a8e77e-9608-41be-9945-2758e26ff4ca" containerName="revoke-keys"
Oct 30 22:34:38.850509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.850491    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:38.901649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.901619    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:38.942951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.942920    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:38.942951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.942954    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:38.943154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.943026    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbqk\" (UniqueName: \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:38.944812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.944773    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:38.944812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.944805    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:38.944974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.944949    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:34:38.946302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:38.946282    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"]
Oct 30 22:34:39.044057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.044025    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.044057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.044057    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.044297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.044113    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbqk\" (UniqueName: \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.044669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.044648    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.046385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.046362    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.097748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.097689    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbqk\" (UniqueName: \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\") pod \"maas-api-revoke-sub-ec843018-6nfvh\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.163134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.163109    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:39.537328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:39.537288    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"]
Oct 30 22:34:39.545163 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:34:39.545138    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53016ee2_c2ec_47ac_b71f_03d478236460.slice/crio-746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a WatchSource:0}: Error finding container 746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a: Status 404 returned error can't find the container with id 746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a
Oct 30 22:34:40.226225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:40.226191    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh" event={"ID":"53016ee2-c2ec-47ac-b71f-03d478236460","Type":"ContainerStarted","Data":"becf9ab02199335dfd480a51b157cc46c8df8a806225f3f6845a2dc1f7c15dde"}
Oct 30 22:34:40.226630 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:40.226249    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh" event={"ID":"53016ee2-c2ec-47ac-b71f-03d478236460","Type":"ContainerStarted","Data":"746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a"}
Oct 30 22:34:40.335831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:40.335757    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh" podStartSLOduration=2.335744139 podStartE2EDuration="2.335744139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:40.335042943 +0000 UTC m=+3089.318949818" watchObservedRunningTime="2026-10-30 22:34:40.335744139 +0000 UTC m=+3089.319651016"
Oct 30 22:34:41.232325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:41.232291    5510 generic.go:356] "Generic (PLEG): container finished" podID="53016ee2-c2ec-47ac-b71f-03d478236460" containerID="becf9ab02199335dfd480a51b157cc46c8df8a806225f3f6845a2dc1f7c15dde" exitCode=0
Oct 30 22:34:41.232325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:41.232326    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh" event={"ID":"53016ee2-c2ec-47ac-b71f-03d478236460","Type":"ContainerDied","Data":"becf9ab02199335dfd480a51b157cc46c8df8a806225f3f6845a2dc1f7c15dde"}
Oct 30 22:34:42.365390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.365358    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:42.476110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.476080    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\" (UniqueName: \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\") pod \"53016ee2-c2ec-47ac-b71f-03d478236460\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") "
Oct 30 22:34:42.476360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.476240    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\") pod \"53016ee2-c2ec-47ac-b71f-03d478236460\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") "
Oct 30 22:34:42.476360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.476276    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\") pod \"53016ee2-c2ec-47ac-b71f-03d478236460\" (UID: \"53016ee2-c2ec-47ac-b71f-03d478236460\") "
Oct 30 22:34:42.476962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.476934    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca" pod "53016ee2-c2ec-47ac-b71f-03d478236460" (UID: "53016ee2-c2ec-47ac-b71f-03d478236460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:42.477089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.477063    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:42.478906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.478813    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk" pod "53016ee2-c2ec-47ac-b71f-03d478236460" (UID: "53016ee2-c2ec-47ac-b71f-03d478236460"). InnerVolumeSpecName "kube-api-access-xdbqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:42.479526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.479496    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca" pod "53016ee2-c2ec-47ac-b71f-03d478236460" (UID: "53016ee2-c2ec-47ac-b71f-03d478236460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:42.578478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.578443    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53016ee2-c2ec-47ac-b71f-03d478236460-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:42.578478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.578474    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdbqk\" (UniqueName: \"kubernetes.io/projected/53016ee2-c2ec-47ac-b71f-03d478236460-kube-api-access-xdbqk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:42.978158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:42.978124    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:34:43.242735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:43.242653    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"
Oct 30 22:34:43.242907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:43.242652    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh" event={"ID":"53016ee2-c2ec-47ac-b71f-03d478236460","Type":"ContainerDied","Data":"746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a"}
Oct 30 22:34:43.242907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:43.242807    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746a7645a3ab80461c85020ff4e3cf1f0d659fad3a9ec7075a9925e179d9157a"
Oct 30 22:34:43.279218 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:34:43.279185    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:34:55.316615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:55.316569    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.035307 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:56.035271    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.063719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:34:56.063670    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:35:06.783254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:06.783218    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:08.058115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:08.058075    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:13.278260 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:35:13.278222    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:35:14.182694 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.182657    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"]
Oct 30 22:35:14.183267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.183244    5510 cpu_manager.go:414] "containerMap: removing container" podUID="53016ee2-c2ec-47ac-b71f-03d478236460" containerName="revoke-keys"
Oct 30 22:35:14.183390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.183270    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53016ee2-c2ec-47ac-b71f-03d478236460" containerName="revoke-keys"
Oct 30 22:35:14.183390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.183389    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="53016ee2-c2ec-47ac-b71f-03d478236460" containerName="revoke-keys"
Oct 30 22:35:14.187495 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.187471    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.238044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.238015    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:14.284102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.284070    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:14.284102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.284097    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:14.284637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.284146    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:35:14.285653 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.285634    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"]
Oct 30 22:35:14.287635 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.287617    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.287709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.287646    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.287709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.287700    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqtk\" (UniqueName: \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.388492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.388453    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqtk\" (UniqueName: \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.388688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.388556    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.388688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.388586    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.389191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.389169    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.390955 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.390934    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.445808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.445741    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqtk\" (UniqueName: \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\") pod \"maas-api-revoke-sub-046e1343-vx422\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.499844 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.499814    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:14.867190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:14.867155    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"]
Oct 30 22:35:14.875000 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:35:14.874971    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4801a85_d793_49a8_b085_a53060e377a4.slice/crio-1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb WatchSource:0}: Error finding container 1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb: Status 404 returned error can't find the container with id 1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb
Oct 30 22:35:15.401124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:15.401088    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422" event={"ID":"c4801a85-d793-49a8-b085-a53060e377a4","Type":"ContainerStarted","Data":"db2c56ed805af893d5a3003585cd41453e8f3c30cb4a3b6e85b15dfdff77ba1d"}
Oct 30 22:35:15.401484 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:15.401145    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422" event={"ID":"c4801a85-d793-49a8-b085-a53060e377a4","Type":"ContainerStarted","Data":"1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb"}
Oct 30 22:35:15.514501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:15.514452    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422" podStartSLOduration=1.514438713 podStartE2EDuration="1.514438713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:15.513236362 +0000 UTC m=+3124.497143238" watchObservedRunningTime="2026-10-30 22:35:15.514438713 +0000 UTC m=+3124.498345588"
Oct 30 22:35:16.407015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:16.406979    5510 generic.go:356] "Generic (PLEG): container finished" podID="c4801a85-d793-49a8-b085-a53060e377a4" containerID="db2c56ed805af893d5a3003585cd41453e8f3c30cb4a3b6e85b15dfdff77ba1d" exitCode=0
Oct 30 22:35:16.407488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:16.407089    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422" event={"ID":"c4801a85-d793-49a8-b085-a53060e377a4","Type":"ContainerDied","Data":"db2c56ed805af893d5a3003585cd41453e8f3c30cb4a3b6e85b15dfdff77ba1d"}
Oct 30 22:35:17.540790 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.540751    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:17.720327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.720294    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\" (UniqueName: \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\") pod \"c4801a85-d793-49a8-b085-a53060e377a4\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") "
Oct 30 22:35:17.720523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.720352    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\") pod \"c4801a85-d793-49a8-b085-a53060e377a4\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") "
Oct 30 22:35:17.720523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.720445    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\") pod \"c4801a85-d793-49a8-b085-a53060e377a4\" (UID: \"c4801a85-d793-49a8-b085-a53060e377a4\") "
Oct 30 22:35:17.720796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.720752    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca" pod "c4801a85-d793-49a8-b085-a53060e377a4" (UID: "c4801a85-d793-49a8-b085-a53060e377a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:17.722541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.722515    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca" pod "c4801a85-d793-49a8-b085-a53060e377a4" (UID: "c4801a85-d793-49a8-b085-a53060e377a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:17.722618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.722544    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk" pod "c4801a85-d793-49a8-b085-a53060e377a4" (UID: "c4801a85-d793-49a8-b085-a53060e377a4"). InnerVolumeSpecName "kube-api-access-pnqtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:17.821455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.821407    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqtk\" (UniqueName: \"kubernetes.io/projected/c4801a85-d793-49a8-b085-a53060e377a4-kube-api-access-pnqtk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:17.821455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.821450    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:17.821455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:17.821461    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4801a85-d793-49a8-b085-a53060e377a4-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.417257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:18.417216    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422" event={"ID":"c4801a85-d793-49a8-b085-a53060e377a4","Type":"ContainerDied","Data":"1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb"}
Oct 30 22:35:18.417257 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:18.417246    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"
Oct 30 22:35:18.417480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:18.417278    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1df7e0afe0008b6a8d5daccc46907657b931cac325bdf3ff50d207884913a9bb"
Oct 30 22:35:21.016992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:21.016952    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:21.684320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:21.684284    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:24.945870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.945838    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"]
Oct 30 22:35:24.946272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.946247    5510 cpu_manager.go:414] "containerMap: removing container" podUID="c4801a85-d793-49a8-b085-a53060e377a4" containerName="revoke-keys"
Oct 30 22:35:24.946272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.946260    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4801a85-d793-49a8-b085-a53060e377a4" containerName="revoke-keys"
Oct 30 22:35:24.946353 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.946335    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4801a85-d793-49a8-b085-a53060e377a4" containerName="revoke-keys"
Oct 30 22:35:24.949738 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.949715    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:24.986839 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.986812    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:24.986997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.986850    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:24.987045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.986981    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6l4\" (UniqueName: \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:24.998259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.998232    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:35:24.998359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:24.998264    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:25.044634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.044593    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:35:25.044634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.044617    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:35:25.046278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.046252    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"]
Oct 30 22:35:25.087746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.087713    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.087939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.087759    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.087939 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.087826    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6l4\" (UniqueName: \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.088476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.088453    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.090173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.090152    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.141699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.141660    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6l4\" (UniqueName: \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\") pod \"maas-api-revoke-sub-d3fba056-4zz7f\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.262825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.262773    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:25.634460 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:25.634370    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"]
Oct 30 22:35:25.642246 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:35:25.642205    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9312669c_09a7_489a_9c13_b1c51bf78237.slice/crio-6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d WatchSource:0}: Error finding container 6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d: Status 404 returned error can't find the container with id 6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d
Oct 30 22:35:26.456048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:26.455966    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f" event={"ID":"9312669c-09a7-489a-9c13-b1c51bf78237","Type":"ContainerStarted","Data":"f154ee06d881dd9f4f812ea2e7df96f75278e46ad8e4073e436a34c7e435e67a"}
Oct 30 22:35:26.456048 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:26.456019    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f" event={"ID":"9312669c-09a7-489a-9c13-b1c51bf78237","Type":"ContainerStarted","Data":"6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d"}
Oct 30 22:35:26.565471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:26.565411    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f" podStartSLOduration=2.56539274 podStartE2EDuration="2.56539274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:26.564619598 +0000 UTC m=+3135.548526476" watchObservedRunningTime="2026-10-30 22:35:26.56539274 +0000 UTC m=+3135.549299617"
Oct 30 22:35:27.183716 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:27.183685    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:35:27.462081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:27.462049    5510 generic.go:356] "Generic (PLEG): container finished" podID="9312669c-09a7-489a-9c13-b1c51bf78237" containerID="f154ee06d881dd9f4f812ea2e7df96f75278e46ad8e4073e436a34c7e435e67a" exitCode=0
Oct 30 22:35:27.462496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:27.462141    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f" event={"ID":"9312669c-09a7-489a-9c13-b1c51bf78237","Type":"ContainerDied","Data":"f154ee06d881dd9f4f812ea2e7df96f75278e46ad8e4073e436a34c7e435e67a"}
Oct 30 22:35:28.946281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:28.945967    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:29.037875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.037809    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\") pod \"9312669c-09a7-489a-9c13-b1c51bf78237\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") "
Oct 30 22:35:29.038109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.037898    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\" (UniqueName: \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\") pod \"9312669c-09a7-489a-9c13-b1c51bf78237\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") "
Oct 30 22:35:29.038109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.037946    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\") pod \"9312669c-09a7-489a-9c13-b1c51bf78237\" (UID: \"9312669c-09a7-489a-9c13-b1c51bf78237\") "
Oct 30 22:35:29.038658 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.038627    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca" pod "9312669c-09a7-489a-9c13-b1c51bf78237" (UID: "9312669c-09a7-489a-9c13-b1c51bf78237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:29.041195 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.041167    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca" pod "9312669c-09a7-489a-9c13-b1c51bf78237" (UID: "9312669c-09a7-489a-9c13-b1c51bf78237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:29.043216 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.043193    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4" pod "9312669c-09a7-489a-9c13-b1c51bf78237" (UID: "9312669c-09a7-489a-9c13-b1c51bf78237"). InnerVolumeSpecName "kube-api-access-gz6l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:29.139725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.139295    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:29.139725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.139330    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6l4\" (UniqueName: \"kubernetes.io/projected/9312669c-09a7-489a-9c13-b1c51bf78237-kube-api-access-gz6l4\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:29.139725 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.139571    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9312669c-09a7-489a-9c13-b1c51bf78237-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:29.513770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.512833    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f" event={"ID":"9312669c-09a7-489a-9c13-b1c51bf78237","Type":"ContainerDied","Data":"6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d"}
Oct 30 22:35:29.513770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.512907    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6087378ac3ce1d8c82f9cf02ee34d55822315073e577c728e3a1713f0433de1d"
Oct 30 22:35:29.513770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:29.512979    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"
Oct 30 22:35:34.785714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:34.785680    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:37.694351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:37.694315    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:35:43.278338 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:35:43.278300    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:35:46.530954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.530904    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:35:46.531384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.530998    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:35:46.620913 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.620873    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-525t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:35:46.621211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.621179    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-525t7" podUID="a5c45657-26b2-4455-8d74-5031505f8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:35:46.667932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.667878    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-ks5ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:35:46.668213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.668167    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-ks5ks" podUID="0f7f5f7a-9377-4e4c-9399-492dedf7a100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:35:46.700507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:46.700293    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:49.319033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:49.319000    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:49.472768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:49.472735    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.334567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:50.334529    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:35:50.449875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:50.449840    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:50.647536 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:35:50.647448    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:00.280098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.280061    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"]
Oct 30 22:36:00.280595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.280499    5510 cpu_manager.go:414] "containerMap: removing container" podUID="9312669c-09a7-489a-9c13-b1c51bf78237" containerName="revoke-keys"
Oct 30 22:36:00.280595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.280512    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9312669c-09a7-489a-9c13-b1c51bf78237" containerName="revoke-keys"
Oct 30 22:36:00.280681 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.280604    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9312669c-09a7-489a-9c13-b1c51bf78237" containerName="revoke-keys"
Oct 30 22:36:00.284087 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.284066    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.334527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.334492    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:00.380343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.380308    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:00.380507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.380427    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:36:00.380507 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.380427    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:00.382111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.382091    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"]
Oct 30 22:36:00.465555 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.465520    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.465741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.465558    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.465741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.465659    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncg8t\" (UniqueName: \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.567007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.566920    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.567007 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.566964    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.567228 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.567012    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncg8t\" (UniqueName: \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.567739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.567716    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.569340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.569320    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.620349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.620307    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncg8t\" (UniqueName: \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\") pod \"maas-api-revoke-sub-ae594979-nvp25\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:00.897119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:00.897031    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:01.264072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:01.264026    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"]
Oct 30 22:36:01.271743 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:36:01.271714    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102303ef_d5c5_4ca4_8f82_dfea7ce75496.slice/crio-d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4 WatchSource:0}: Error finding container d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4: Status 404 returned error can't find the container with id d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4
Oct 30 22:36:01.697156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:01.697114    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25" event={"ID":"102303ef-d5c5-4ca4-8f82-dfea7ce75496","Type":"ContainerStarted","Data":"bc0121ae082dec0abfb20a233c368df7f5ee5abc09eb9bc0caf031112619347b"}
Oct 30 22:36:01.697553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:01.697179    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25" event={"ID":"102303ef-d5c5-4ca4-8f82-dfea7ce75496","Type":"ContainerStarted","Data":"d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4"}
Oct 30 22:36:01.803541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:01.803489    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25" podStartSLOduration=1.803474317 podStartE2EDuration="1.803474317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:01.802817664 +0000 UTC m=+3170.786724535" watchObservedRunningTime="2026-10-30 22:36:01.803474317 +0000 UTC m=+3170.787381193"
Oct 30 22:36:02.703647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:02.703612    5510 generic.go:356] "Generic (PLEG): container finished" podID="102303ef-d5c5-4ca4-8f82-dfea7ce75496" containerID="bc0121ae082dec0abfb20a233c368df7f5ee5abc09eb9bc0caf031112619347b" exitCode=0
Oct 30 22:36:02.704116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:02.703702    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25" event={"ID":"102303ef-d5c5-4ca4-8f82-dfea7ce75496","Type":"ContainerDied","Data":"bc0121ae082dec0abfb20a233c368df7f5ee5abc09eb9bc0caf031112619347b"}
Oct 30 22:36:03.839571 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.839544    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:03.899514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.899482    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\" (UniqueName: \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\") pod \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") "
Oct 30 22:36:03.899689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.899547    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\") pod \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") "
Oct 30 22:36:03.899689 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.899622    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\") pod \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\" (UID: \"102303ef-d5c5-4ca4-8f82-dfea7ce75496\") "
Oct 30 22:36:03.899963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.899939    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca" pod "102303ef-d5c5-4ca4-8f82-dfea7ce75496" (UID: "102303ef-d5c5-4ca4-8f82-dfea7ce75496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:03.901646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.901623    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca" pod "102303ef-d5c5-4ca4-8f82-dfea7ce75496" (UID: "102303ef-d5c5-4ca4-8f82-dfea7ce75496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:03.901820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:03.901672    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t" pod "102303ef-d5c5-4ca4-8f82-dfea7ce75496" (UID: "102303ef-d5c5-4ca4-8f82-dfea7ce75496"). InnerVolumeSpecName "kube-api-access-ncg8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:04.001077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.001004    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncg8t\" (UniqueName: \"kubernetes.io/projected/102303ef-d5c5-4ca4-8f82-dfea7ce75496-kube-api-access-ncg8t\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:04.001077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.001029    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:04.001077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.001039    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102303ef-d5c5-4ca4-8f82-dfea7ce75496-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:04.714261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.714233    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"
Oct 30 22:36:04.714420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.714234    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25" event={"ID":"102303ef-d5c5-4ca4-8f82-dfea7ce75496","Type":"ContainerDied","Data":"d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4"}
Oct 30 22:36:04.714420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:04.714352    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0898be1b5d6a69338fea3a1e96ab340cfa7f1f4fc059303fad7d625a41878a4"
Oct 30 22:36:10.371671 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:10.371625    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:10.759005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:10.758958    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:13.154687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:13.154653    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:36:13.279581 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:36:13.279545    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:36:25.157118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:25.157084    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:36:32.033128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.033092    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"]
Oct 30 22:36:32.033618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.033521    5510 cpu_manager.go:414] "containerMap: removing container" podUID="102303ef-d5c5-4ca4-8f82-dfea7ce75496" containerName="revoke-keys"
Oct 30 22:36:32.033618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.033534    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102303ef-d5c5-4ca4-8f82-dfea7ce75496" containerName="revoke-keys"
Oct 30 22:36:32.033696 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.033641    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="102303ef-d5c5-4ca4-8f82-dfea7ce75496" containerName="revoke-keys"
Oct 30 22:36:32.036714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.036693    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.090971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.090942    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:36:32.134302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.134262    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:36:32.134302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.134267    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:36:32.134557 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.134426    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:36:32.136096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.136069    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"]
Oct 30 22:36:32.181937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.181894    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.182145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.182010    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l76\" (UniqueName: \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.182211 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.182147    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.283273 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.283174    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.283469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.283272    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.283469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.283314    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l76\" (UniqueName: \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.284039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.284009    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.286210 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.286177    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.336774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.336742    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l76\" (UniqueName: \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\") pod \"maas-api-revoke-sub-a4aef7d7-bhclr\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.350394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.350375    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:32.718458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.718426    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"]
Oct 30 22:36:32.725569 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:36:32.725540    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159a4f65_272e_4f41_9a10_0f8d8970082b.slice/crio-0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8 WatchSource:0}: Error finding container 0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8: Status 404 returned error can't find the container with id 0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8
Oct 30 22:36:32.853958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:32.853926    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr" event={"ID":"159a4f65-272e-4f41-9a10-0f8d8970082b","Type":"ContainerStarted","Data":"0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8"}
Oct 30 22:36:33.860340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:33.860246    5510 generic.go:356] "Generic (PLEG): container finished" podID="159a4f65-272e-4f41-9a10-0f8d8970082b" containerID="5a03d7b1397afabf6d7c3d21f9046a7af6984125ba7085c40a3a4219c4890303" exitCode=0
Oct 30 22:36:33.860767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:33.860327    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr" event={"ID":"159a4f65-272e-4f41-9a10-0f8d8970082b","Type":"ContainerDied","Data":"5a03d7b1397afabf6d7c3d21f9046a7af6984125ba7085c40a3a4219c4890303"}
Oct 30 22:36:34.985570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:34.985543    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:35.059217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.059186    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:35.110250 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.110210    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\") pod \"159a4f65-272e-4f41-9a10-0f8d8970082b\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") "
Oct 30 22:36:35.110441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.110293    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\") pod \"159a4f65-272e-4f41-9a10-0f8d8970082b\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") "
Oct 30 22:36:35.110516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.110448    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\" (UniqueName: \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\") pod \"159a4f65-272e-4f41-9a10-0f8d8970082b\" (UID: \"159a4f65-272e-4f41-9a10-0f8d8970082b\") "
Oct 30 22:36:35.110669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.110642    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca" pod "159a4f65-272e-4f41-9a10-0f8d8970082b" (UID: "159a4f65-272e-4f41-9a10-0f8d8970082b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:35.110750 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.110732    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:35.112259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.112226    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca" pod "159a4f65-272e-4f41-9a10-0f8d8970082b" (UID: "159a4f65-272e-4f41-9a10-0f8d8970082b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:35.112441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.112419    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76" pod "159a4f65-272e-4f41-9a10-0f8d8970082b" (UID: "159a4f65-272e-4f41-9a10-0f8d8970082b"). InnerVolumeSpecName "kube-api-access-v7l76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:35.212051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.212026    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7l76\" (UniqueName: \"kubernetes.io/projected/159a4f65-272e-4f41-9a10-0f8d8970082b-kube-api-access-v7l76\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:35.212051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.212051    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159a4f65-272e-4f41-9a10-0f8d8970082b-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:35.870098 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.870069    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"
Oct 30 22:36:35.870277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.870070    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr" event={"ID":"159a4f65-272e-4f41-9a10-0f8d8970082b","Type":"ContainerDied","Data":"0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8"}
Oct 30 22:36:35.870277 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:35.870213    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f971690026b95dc2bd172f49ed65040df35c4d0414e42ce0fbf57e4eb1454b8"
Oct 30 22:36:43.279560 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:36:43.279508    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:36:44.862943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:44.862907    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:36:46.049617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:46.049583    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"]
Oct 30 22:36:46.095455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:46.095412    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec15aa31-tf7w9"]
Oct 30 22:36:47.123989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:47.123952    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58" path="/var/lib/kubelet/pods/d3b4ebdd-ae00-41ba-b4e0-0530b7c97c58/volumes"
Oct 30 22:36:57.284727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:36:57.284685    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:37:02.625660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.625618    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"]
Oct 30 22:37:02.626322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.626263    5510 cpu_manager.go:414] "containerMap: removing container" podUID="159a4f65-272e-4f41-9a10-0f8d8970082b" containerName="revoke-keys"
Oct 30 22:37:02.626322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.626287    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159a4f65-272e-4f41-9a10-0f8d8970082b" containerName="revoke-keys"
Oct 30 22:37:02.626452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.626399    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="159a4f65-272e-4f41-9a10-0f8d8970082b" containerName="revoke-keys"
Oct 30 22:37:02.629754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.629728    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.701871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.701828    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlkq\" (UniqueName: \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.702080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.701883    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.702080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.702031    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.727287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.727253    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:02.727287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.727275    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:02.727498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.727254    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:02.727498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.727308    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:37:02.728998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.728978    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"]
Oct 30 22:37:02.802855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.802811    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlkq\" (UniqueName: \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.802855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.802854    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.803085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.802902    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.803487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.803468    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.805287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.805268    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.857702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.857665    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlkq\" (UniqueName: \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\") pod \"maas-api-revoke-sub-79805ca9-57j92\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:02.942849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:02.942722    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:03.315714 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:03.315669    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"]
Oct 30 22:37:03.323521 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:37:03.323491    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3152c2e_e002_42ad_9c2d_14d281730df4.slice/crio-76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483 WatchSource:0}: Error finding container 76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483: Status 404 returned error can't find the container with id 76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483
Oct 30 22:37:04.009816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:04.009755    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92" event={"ID":"b3152c2e-e002-42ad-9c2d-14d281730df4","Type":"ContainerStarted","Data":"be695aa2201890b9e5bca0e7d83ce8ca2a6c0e32be1188aa288947bfb531f8c0"}
Oct 30 22:37:04.010225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:04.009836    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92" event={"ID":"b3152c2e-e002-42ad-9c2d-14d281730df4","Type":"ContainerStarted","Data":"76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483"}
Oct 30 22:37:04.123388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:04.123330    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92" podStartSLOduration=2.123317247 podStartE2EDuration="2.123317247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:04.120583776 +0000 UTC m=+3233.104490649" watchObservedRunningTime="2026-10-30 22:37:04.123317247 +0000 UTC m=+3233.107224136"
Oct 30 22:37:05.015503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:05.015467    5510 generic.go:356] "Generic (PLEG): container finished" podID="b3152c2e-e002-42ad-9c2d-14d281730df4" containerID="be695aa2201890b9e5bca0e7d83ce8ca2a6c0e32be1188aa288947bfb531f8c0" exitCode=0
Oct 30 22:37:05.015941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:05.015552    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92" event={"ID":"b3152c2e-e002-42ad-9c2d-14d281730df4","Type":"ContainerDied","Data":"be695aa2201890b9e5bca0e7d83ce8ca2a6c0e32be1188aa288947bfb531f8c0"}
Oct 30 22:37:06.156424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.156394    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:06.237086 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.237049    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\" (UniqueName: \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\") pod \"b3152c2e-e002-42ad-9c2d-14d281730df4\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") "
Oct 30 22:37:06.237294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.237121    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\") pod \"b3152c2e-e002-42ad-9c2d-14d281730df4\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") "
Oct 30 22:37:06.237294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.237142    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\") pod \"b3152c2e-e002-42ad-9c2d-14d281730df4\" (UID: \"b3152c2e-e002-42ad-9c2d-14d281730df4\") "
Oct 30 22:37:06.237539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.237515    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca" pod "b3152c2e-e002-42ad-9c2d-14d281730df4" (UID: "b3152c2e-e002-42ad-9c2d-14d281730df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:06.239263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.239238    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq" pod "b3152c2e-e002-42ad-9c2d-14d281730df4" (UID: "b3152c2e-e002-42ad-9c2d-14d281730df4"). InnerVolumeSpecName "kube-api-access-7mlkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:06.239332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.239244    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca" pod "b3152c2e-e002-42ad-9c2d-14d281730df4" (UID: "b3152c2e-e002-42ad-9c2d-14d281730df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:06.337739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.337653    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlkq\" (UniqueName: \"kubernetes.io/projected/b3152c2e-e002-42ad-9c2d-14d281730df4-kube-api-access-7mlkq\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:06.337739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.337684    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:06.337739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:06.337697    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3152c2e-e002-42ad-9c2d-14d281730df4-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:07.025718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:07.025685    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"
Oct 30 22:37:07.025718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:07.025709    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92" event={"ID":"b3152c2e-e002-42ad-9c2d-14d281730df4","Type":"ContainerDied","Data":"76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483"}
Oct 30 22:37:07.025946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:07.025752    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a31d08f7f0418708b60195b9ef52bf45641d76a2e1c4024580350646529483"
Oct 30 22:37:13.279225 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:37:13.279179    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:37:17.173459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:17.173425    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:37:26.437082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.437009    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"]
Oct 30 22:37:26.437450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.437403    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b3152c2e-e002-42ad-9c2d-14d281730df4" containerName="revoke-keys"
Oct 30 22:37:26.437450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.437416    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3152c2e-e002-42ad-9c2d-14d281730df4" containerName="revoke-keys"
Oct 30 22:37:26.437540 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.437493    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3152c2e-e002-42ad-9c2d-14d281730df4" containerName="revoke-keys"
Oct 30 22:37:26.440858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.440835    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.493434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.493397    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"]
Oct 30 22:37:26.537125 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.537098    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:26.537298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.537141    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:26.537298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.537154    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:26.537298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.537185    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:37:26.562260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.562233    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsjw\" (UniqueName: \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.562380 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.562289    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.562433 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.562385    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.663134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.663100    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.663318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.663148    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.663318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.663202    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsjw\" (UniqueName: \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.663744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.663721    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.665480 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.665456    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.716450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.716422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsjw\" (UniqueName: \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\") pod \"maas-api-revoke-sub-0b7e8edb-7vmps\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:26.756122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:26.756095    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:27.130835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:27.130806    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"]
Oct 30 22:37:27.132026 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:37:27.131999    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0394ed3a_8ee7_446a_9f48_48c61d2ea36a.slice/crio-2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed WatchSource:0}: Error finding container 2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed: Status 404 returned error can't find the container with id 2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed
Oct 30 22:37:27.153610 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:27.153578    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps" event={"ID":"0394ed3a-8ee7-446a-9f48-48c61d2ea36a","Type":"ContainerStarted","Data":"2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed"}
Oct 30 22:37:28.160343 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:28.160306    5510 generic.go:356] "Generic (PLEG): container finished" podID="0394ed3a-8ee7-446a-9f48-48c61d2ea36a" containerID="a9f0d1f8d07544fd73f21df19abe9682ab6cbb398d7fae33a41ac656a3bf8540" exitCode=0
Oct 30 22:37:28.160811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:28.160350    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps" event={"ID":"0394ed3a-8ee7-446a-9f48-48c61d2ea36a","Type":"ContainerDied","Data":"a9f0d1f8d07544fd73f21df19abe9682ab6cbb398d7fae33a41ac656a3bf8540"}
Oct 30 22:37:29.291368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.291342    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:29.388501 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.388467    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\" (UniqueName: \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\") pod \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") "
Oct 30 22:37:29.388665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.388602    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\") pod \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") "
Oct 30 22:37:29.388665 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.388648    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\") pod \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\" (UID: \"0394ed3a-8ee7-446a-9f48-48c61d2ea36a\") "
Oct 30 22:37:29.389052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.389028    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca" pod "0394ed3a-8ee7-446a-9f48-48c61d2ea36a" (UID: "0394ed3a-8ee7-446a-9f48-48c61d2ea36a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:29.390602 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.390578    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca" pod "0394ed3a-8ee7-446a-9f48-48c61d2ea36a" (UID: "0394ed3a-8ee7-446a-9f48-48c61d2ea36a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:29.390692 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.390612    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw" pod "0394ed3a-8ee7-446a-9f48-48c61d2ea36a" (UID: "0394ed3a-8ee7-446a-9f48-48c61d2ea36a"). InnerVolumeSpecName "kube-api-access-fqsjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:29.489869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.489838    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqsjw\" (UniqueName: \"kubernetes.io/projected/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-kube-api-access-fqsjw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:29.489869 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.489867    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:29.490072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:29.489877    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0394ed3a-8ee7-446a-9f48-48c61d2ea36a-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:30.170389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:30.170361    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"
Oct 30 22:37:30.170570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:30.170356    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps" event={"ID":"0394ed3a-8ee7-446a-9f48-48c61d2ea36a","Type":"ContainerDied","Data":"2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed"}
Oct 30 22:37:30.170570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:30.170484    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8ae4d4a51381f1bf783b7a6a9e7d3b19a8e6ab91c71659e0a33aa66e0693ed"
Oct 30 22:37:35.400976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:35.400930    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.734145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.734106    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"]
Oct 30 22:37:37.734803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.734764    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0394ed3a-8ee7-446a-9f48-48c61d2ea36a" containerName="revoke-keys"
Oct 30 22:37:37.734861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.734806    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0394ed3a-8ee7-446a-9f48-48c61d2ea36a" containerName="revoke-keys"
Oct 30 22:37:37.734930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.734918    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0394ed3a-8ee7-446a-9f48-48c61d2ea36a" containerName="revoke-keys"
Oct 30 22:37:37.738756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.738734    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.761156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.761118    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqj7\" (UniqueName: \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.761305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.761186    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.761387 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.761309    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.789559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.789532    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:37.835383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.835344    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:37.835544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.835402    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:37:37.835544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.835441    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:37.837217 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.837198    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"]
Oct 30 22:37:37.862595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.862569    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqj7\" (UniqueName: \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.862710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.862616    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.862710 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.862684    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.863595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.863572    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.864974 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.864951    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:37.916342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:37.916315    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqj7\" (UniqueName: \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\") pod \"maas-api-revoke-sub-3d4156f1-52jkx\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:38.054577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:38.054495    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:38.431609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:38.431528    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"]
Oct 30 22:37:38.438980 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:37:38.438949    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda169f2b0_160b_4733_a336_a57d6b1b662c.slice/crio-8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714 WatchSource:0}: Error finding container 8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714: Status 404 returned error can't find the container with id 8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714
Oct 30 22:37:39.215357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:39.215323    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx" event={"ID":"a169f2b0-160b-4733-a336-a57d6b1b662c","Type":"ContainerStarted","Data":"8f2a4d5164b0415de699a3ec24748abed434d56b410c1d3f3e2038ab969e2f14"}
Oct 30 22:37:39.215761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:39.215376    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx" event={"ID":"a169f2b0-160b-4733-a336-a57d6b1b662c","Type":"ContainerStarted","Data":"8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714"}
Oct 30 22:37:39.240485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:39.240448    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:37:39.345012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:39.344954    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx" podStartSLOduration=2.344940055 podStartE2EDuration="2.344940055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:39.343820132 +0000 UTC m=+3268.327727005" watchObservedRunningTime="2026-10-30 22:37:39.344940055 +0000 UTC m=+3268.328846928"
Oct 30 22:37:40.235938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:40.235895    5510 generic.go:356] "Generic (PLEG): container finished" podID="a169f2b0-160b-4733-a336-a57d6b1b662c" containerID="8f2a4d5164b0415de699a3ec24748abed434d56b410c1d3f3e2038ab969e2f14" exitCode=0
Oct 30 22:37:40.236398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:40.235957    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx" event={"ID":"a169f2b0-160b-4733-a336-a57d6b1b662c","Type":"ContainerDied","Data":"8f2a4d5164b0415de699a3ec24748abed434d56b410c1d3f3e2038ab969e2f14"}
Oct 30 22:37:41.414114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.414089    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:41.432736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.432708    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\") pod \"a169f2b0-160b-4733-a336-a57d6b1b662c\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") "
Oct 30 22:37:41.432888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.432750    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\" (UniqueName: \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\") pod \"a169f2b0-160b-4733-a336-a57d6b1b662c\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") "
Oct 30 22:37:41.432888 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.432852    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\") pod \"a169f2b0-160b-4733-a336-a57d6b1b662c\" (UID: \"a169f2b0-160b-4733-a336-a57d6b1b662c\") "
Oct 30 22:37:41.433119 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.433093    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca" pod "a169f2b0-160b-4733-a336-a57d6b1b662c" (UID: "a169f2b0-160b-4733-a336-a57d6b1b662c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:41.433214 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.433196    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:41.434946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.434921    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7" pod "a169f2b0-160b-4733-a336-a57d6b1b662c" (UID: "a169f2b0-160b-4733-a336-a57d6b1b662c"). InnerVolumeSpecName "kube-api-access-8kqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:41.435071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.434969    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca" pod "a169f2b0-160b-4733-a336-a57d6b1b662c" (UID: "a169f2b0-160b-4733-a336-a57d6b1b662c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:41.533997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.533911    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a169f2b0-160b-4733-a336-a57d6b1b662c-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:41.533997 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:41.533936    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqj7\" (UniqueName: \"kubernetes.io/projected/a169f2b0-160b-4733-a336-a57d6b1b662c-kube-api-access-8kqj7\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:42.248879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:42.248849    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"
Oct 30 22:37:42.249078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:42.248840    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx" event={"ID":"a169f2b0-160b-4733-a336-a57d6b1b662c","Type":"ContainerDied","Data":"8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714"}
Oct 30 22:37:42.249078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:42.248983    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3c95651a2775c1b14a32d0367d5834d972a35885cbe658d3638299f35b6714"
Oct 30 22:37:43.278837 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:37:43.278800    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:37:49.050965 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:49.050921    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:49.747675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:49.747625    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:50.549528 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:37:50.549493    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:38:01.778003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:01.777958    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:38:08.914813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:08.914768    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"]
Oct 30 22:38:08.960568 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:08.960531    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-52jkx"]
Oct 30 22:38:09.078754 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.078712    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"]
Oct 30 22:38:09.079347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.079327    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a169f2b0-160b-4733-a336-a57d6b1b662c" containerName="revoke-keys"
Oct 30 22:38:09.079398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.079351    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a169f2b0-160b-4733-a336-a57d6b1b662c" containerName="revoke-keys"
Oct 30 22:38:09.079530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.079518    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a169f2b0-160b-4733-a336-a57d6b1b662c" containerName="revoke-keys"
Oct 30 22:38:09.083425 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.083395    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.125147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.125095    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a169f2b0-160b-4733-a336-a57d6b1b662c" path="/var/lib/kubelet/pods/a169f2b0-160b-4733-a336-a57d6b1b662c/volumes"
Oct 30 22:38:09.141643 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.141614    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"]
Oct 30 22:38:09.144526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.144493    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:09.144685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.144526    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:38:09.145029 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.145008    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:09.172999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.172918    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc7f\" (UniqueName: \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.173154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.173068    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.173236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.173211    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.184333 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.184311    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:09.273801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.273757    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.274015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.273890    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.274015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.273946    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc7f\" (UniqueName: \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.274530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.274505    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.276194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.276171    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.328614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.328582    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc7f\" (UniqueName: \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\") pod \"maas-api-revoke-sub-3d4156f1-5xmxd\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.400315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.400276    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:09.770707 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:09.770666    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"]
Oct 30 22:38:09.778186 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:38:09.778158    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355220cd_e662_4c42_aecd_e69f875c507f.slice/crio-6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784 WatchSource:0}: Error finding container 6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784: Status 404 returned error can't find the container with id 6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784
Oct 30 22:38:10.439773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:10.439682    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd" event={"ID":"355220cd-e662-4c42-aecd-e69f875c507f","Type":"ContainerStarted","Data":"528e443233394efe0542545e03ca66fca2ff8893a2ca6342705ae279bf46af56"}
Oct 30 22:38:10.439773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:10.439732    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd" event={"ID":"355220cd-e662-4c42-aecd-e69f875c507f","Type":"ContainerStarted","Data":"6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784"}
Oct 30 22:38:10.549144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:10.549088    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd" podStartSLOduration=1.549073878 podStartE2EDuration="1.549073878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:10.547521912 +0000 UTC m=+3299.531428792" watchObservedRunningTime="2026-10-30 22:38:10.549073878 +0000 UTC m=+3299.532980751"
Oct 30 22:38:11.446826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:11.446763    5510 generic.go:356] "Generic (PLEG): container finished" podID="355220cd-e662-4c42-aecd-e69f875c507f" containerID="528e443233394efe0542545e03ca66fca2ff8893a2ca6342705ae279bf46af56" exitCode=0
Oct 30 22:38:11.447222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:11.446836    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd" event={"ID":"355220cd-e662-4c42-aecd-e69f875c507f","Type":"ContainerDied","Data":"528e443233394efe0542545e03ca66fca2ff8893a2ca6342705ae279bf46af56"}
Oct 30 22:38:11.951819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:11.950615    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:38:12.392632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.392604    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:38:12.582482 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.582458    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:12.596632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.596613    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\") pod \"355220cd-e662-4c42-aecd-e69f875c507f\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") "
Oct 30 22:38:12.596729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.596647    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\" (UniqueName: \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\") pod \"355220cd-e662-4c42-aecd-e69f875c507f\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") "
Oct 30 22:38:12.596729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.596694    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\") pod \"355220cd-e662-4c42-aecd-e69f875c507f\" (UID: \"355220cd-e662-4c42-aecd-e69f875c507f\") "
Oct 30 22:38:12.596987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.596961    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca" pod "355220cd-e662-4c42-aecd-e69f875c507f" (UID: "355220cd-e662-4c42-aecd-e69f875c507f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:12.598582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.598558    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f" pod "355220cd-e662-4c42-aecd-e69f875c507f" (UID: "355220cd-e662-4c42-aecd-e69f875c507f"). InnerVolumeSpecName "kube-api-access-hdc7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:12.598676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.598608    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca" pod "355220cd-e662-4c42-aecd-e69f875c507f" (UID: "355220cd-e662-4c42-aecd-e69f875c507f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:12.697435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.697361    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:12.697435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.697393    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355220cd-e662-4c42-aecd-e69f875c507f-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:12.697435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:12.697408    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdc7f\" (UniqueName: \"kubernetes.io/projected/355220cd-e662-4c42-aecd-e69f875c507f-kube-api-access-hdc7f\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:13.278874 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:38:13.278840    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:38:13.457235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:13.457161    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"
Oct 30 22:38:13.457235 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:13.457198    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd" event={"ID":"355220cd-e662-4c42-aecd-e69f875c507f","Type":"ContainerDied","Data":"6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784"}
Oct 30 22:38:13.457424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:13.457245    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d608a30500a8e1fac2c299c9490aca1f40f2d8ea599a38e86c4e14aaadd4784"
Oct 30 22:38:23.394169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:23.394133    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.720149 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:23.720120    5510 scope.go:122] "RemoveContainer" containerID="e38ac69ac06121537f662d4dec05f7ab72392ca5d9723c4f9af72cd4792dbb49"
Oct 30 22:38:35.777581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:38:35.777533    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.278353 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:38:43.278306    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:39:01.842794 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:01.842697    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:39:13.278617 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:39:13.278574    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:39:30.045506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.045462    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"]
Oct 30 22:39:30.092169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.092139    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cvhgd"]
Oct 30 22:39:30.801179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.801141    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"]
Oct 30 22:39:30.801647 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.801627    5510 cpu_manager.go:414] "containerMap: removing container" podUID="355220cd-e662-4c42-aecd-e69f875c507f" containerName="revoke-keys"
Oct 30 22:39:30.801748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.801649    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355220cd-e662-4c42-aecd-e69f875c507f" containerName="revoke-keys"
Oct 30 22:39:30.801842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.801797    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="355220cd-e662-4c42-aecd-e69f875c507f" containerName="revoke-keys"
Oct 30 22:39:30.804856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.804835    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.856905 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.856861    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:30.874561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.874531    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47mw\" (UniqueName: \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.874726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.874574    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.874726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.874709    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.902745 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.902712    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:30.902884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.902829    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:30.903486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.903458    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:39:30.907492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.907466    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"]
Oct 30 22:39:30.975810 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.975766    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.975927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.975843    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47mw\" (UniqueName: \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.975927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.975866    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.976509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.976490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:30.978075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:30.978053    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:31.029691 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.029656    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47mw\" (UniqueName: \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\") pod \"maas-api-revoke-sub-e2fc9c5b-jlbw6\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:31.117564 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.117479    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:31.123932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.123902    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a8e77e-9608-41be-9945-2758e26ff4ca" path="/var/lib/kubelet/pods/b2a8e77e-9608-41be-9945-2758e26ff4ca/volumes"
Oct 30 22:39:31.485802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.485750    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"]
Oct 30 22:39:31.493276 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:39:31.493243    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96164e39_9322_409b_b9c7_f79f05405da8.slice/crio-1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4 WatchSource:0}: Error finding container 1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4: Status 404 returned error can't find the container with id 1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4
Oct 30 22:39:31.849414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.849379    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6" event={"ID":"96164e39-9322-409b-b9c7-f79f05405da8","Type":"ContainerStarted","Data":"843945958d8dfb7b6dde23d8ea4d6d3bf717504a9780cb91434dc2965549e093"}
Oct 30 22:39:31.849591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.849432    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6" event={"ID":"96164e39-9322-409b-b9c7-f79f05405da8","Type":"ContainerStarted","Data":"1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4"}
Oct 30 22:39:31.956494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:31.956444    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6" podStartSLOduration=1.9564298519999999 podStartE2EDuration="1.956429852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:31.954967998 +0000 UTC m=+3380.938874874" watchObservedRunningTime="2026-10-30 22:39:31.956429852 +0000 UTC m=+3380.940336724"
Oct 30 22:39:32.856261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:32.856231    5510 generic.go:356] "Generic (PLEG): container finished" podID="96164e39-9322-409b-b9c7-f79f05405da8" containerID="843945958d8dfb7b6dde23d8ea4d6d3bf717504a9780cb91434dc2965549e093" exitCode=0
Oct 30 22:39:32.856721 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:32.856315    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6" event={"ID":"96164e39-9322-409b-b9c7-f79f05405da8","Type":"ContainerDied","Data":"843945958d8dfb7b6dde23d8ea4d6d3bf717504a9780cb91434dc2965549e093"}
Oct 30 22:39:33.997397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:33.997373    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:34.105292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.105260    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\" (UniqueName: \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\") pod \"96164e39-9322-409b-b9c7-f79f05405da8\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") "
Oct 30 22:39:34.105502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.105376    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\") pod \"96164e39-9322-409b-b9c7-f79f05405da8\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") "
Oct 30 22:39:34.105502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.105412    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\") pod \"96164e39-9322-409b-b9c7-f79f05405da8\" (UID: \"96164e39-9322-409b-b9c7-f79f05405da8\") "
Oct 30 22:39:34.105767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.105740    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca" pod "96164e39-9322-409b-b9c7-f79f05405da8" (UID: "96164e39-9322-409b-b9c7-f79f05405da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:34.107359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.107334    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw" pod "96164e39-9322-409b-b9c7-f79f05405da8" (UID: "96164e39-9322-409b-b9c7-f79f05405da8"). InnerVolumeSpecName "kube-api-access-t47mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:34.107438 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.107411    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca" pod "96164e39-9322-409b-b9c7-f79f05405da8" (UID: "96164e39-9322-409b-b9c7-f79f05405da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:34.206079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.206047    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:34.206079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.206075    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96164e39-9322-409b-b9c7-f79f05405da8-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:34.206079 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.206085    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t47mw\" (UniqueName: \"kubernetes.io/projected/96164e39-9322-409b-b9c7-f79f05405da8-kube-api-access-t47mw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:34.867074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.867039    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"
Oct 30 22:39:34.867268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.867071    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6" event={"ID":"96164e39-9322-409b-b9c7-f79f05405da8","Type":"ContainerDied","Data":"1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4"}
Oct 30 22:39:34.867268 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:34.867154    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1767298a66158b1ac1765a53773c40af1fd61a2006811eb9eb56c28e936cbdc4"
Oct 30 22:39:36.093259 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:36.093224    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:41.576518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.576471    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"]
Oct 30 22:39:41.577046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.576988    5510 cpu_manager.go:414] "containerMap: removing container" podUID="96164e39-9322-409b-b9c7-f79f05405da8" containerName="revoke-keys"
Oct 30 22:39:41.577046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.577005    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96164e39-9322-409b-b9c7-f79f05405da8" containerName="revoke-keys"
Oct 30 22:39:41.577182 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.577134    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="96164e39-9322-409b-b9c7-f79f05405da8" containerName="revoke-keys"
Oct 30 22:39:41.580506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.580485    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.634026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.633992    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"]
Oct 30 22:39:41.649796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.649753    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:41.672856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.672825    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kng7b\" (UniqueName: \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.673026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.672892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.673026 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.673012    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.677397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.677345    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:39:41.677397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.677360    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:41.677538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.677401    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:41.773773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.773748    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kng7b\" (UniqueName: \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.773926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.773889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.773995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.773926    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.774800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.774756    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.776156 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.776137    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.828278 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.828224    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kng7b\" (UniqueName: \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\") pod \"maas-api-revoke-sub-5a29efb3-xrg6l\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:41.893516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:41.893485    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:42.263412 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:42.263377    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"]
Oct 30 22:39:42.270797 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:39:42.270756    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a4f229_055c_43b7_a1b9_618a9f575ef2.slice/crio-1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3 WatchSource:0}: Error finding container 1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3: Status 404 returned error can't find the container with id 1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3
Oct 30 22:39:42.907385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:42.907303    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l" event={"ID":"34a4f229-055c-43b7-a1b9-618a9f575ef2","Type":"ContainerStarted","Data":"1063a6864fac8388e748493290e07f1e6146eb886972da84a79f8d2c6df0eaba"}
Oct 30 22:39:42.907385 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:42.907355    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l" event={"ID":"34a4f229-055c-43b7-a1b9-618a9f575ef2","Type":"ContainerStarted","Data":"1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3"}
Oct 30 22:39:43.015060 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.015004    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l" podStartSLOduration=2.014980765 podStartE2EDuration="2.014980765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:43.01358923 +0000 UTC m=+3391.997496106" watchObservedRunningTime="2026-10-30 22:39:43.014980765 +0000 UTC m=+3391.998887641"
Oct 30 22:39:43.063814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.063737    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"]
Oct 30 22:39:43.065331 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.065309    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-6nfvh"]
Oct 30 22:39:43.123471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.123440    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53016ee2-c2ec-47ac-b71f-03d478236460" path="/var/lib/kubelet/pods/53016ee2-c2ec-47ac-b71f-03d478236460/volumes"
Oct 30 22:39:43.279396 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:39:43.279355    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:39:43.690541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.690507    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:39:43.914148 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.914114    5510 generic.go:356] "Generic (PLEG): container finished" podID="34a4f229-055c-43b7-a1b9-618a9f575ef2" containerID="1063a6864fac8388e748493290e07f1e6146eb886972da84a79f8d2c6df0eaba" exitCode=0
Oct 30 22:39:43.914549 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:43.914197    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l" event={"ID":"34a4f229-055c-43b7-a1b9-618a9f575ef2","Type":"ContainerDied","Data":"1063a6864fac8388e748493290e07f1e6146eb886972da84a79f8d2c6df0eaba"}
Oct 30 22:39:45.401800 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.399936    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:45.447820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.447763    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\" (UniqueName: \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\") pod \"34a4f229-055c-43b7-a1b9-618a9f575ef2\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") "
Oct 30 22:39:45.447990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.447826    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\") pod \"34a4f229-055c-43b7-a1b9-618a9f575ef2\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") "
Oct 30 22:39:45.447990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.447873    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\") pod \"34a4f229-055c-43b7-a1b9-618a9f575ef2\" (UID: \"34a4f229-055c-43b7-a1b9-618a9f575ef2\") "
Oct 30 22:39:45.448437 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.448404    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca" pod "34a4f229-055c-43b7-a1b9-618a9f575ef2" (UID: "34a4f229-055c-43b7-a1b9-618a9f575ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:45.451225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.451198    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b" pod "34a4f229-055c-43b7-a1b9-618a9f575ef2" (UID: "34a4f229-055c-43b7-a1b9-618a9f575ef2"). InnerVolumeSpecName "kube-api-access-kng7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:45.479804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.475892    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca" pod "34a4f229-055c-43b7-a1b9-618a9f575ef2" (UID: "34a4f229-055c-43b7-a1b9-618a9f575ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:45.548511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.548450    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.548511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.548478    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kng7b\" (UniqueName: \"kubernetes.io/projected/34a4f229-055c-43b7-a1b9-618a9f575ef2-kube-api-access-kng7b\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.548511 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.548488    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a4f229-055c-43b7-a1b9-618a9f575ef2-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.934792 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.934738    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l" event={"ID":"34a4f229-055c-43b7-a1b9-618a9f575ef2","Type":"ContainerDied","Data":"1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3"}
Oct 30 22:39:45.935012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.934855    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1816b6061fd2d1a6397221302299f3d067c739553ea83055c556d683032af6d3"
Oct 30 22:39:45.935012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:45.934760    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"
Oct 30 22:39:46.531949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:46.531900    5510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8f4kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:39:46.532393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:46.531961    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8f4kg" podUID="3a829d55-09d6-4110-ab64-7b84c761319f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:39:54.272417 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:54.272385    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:39:58.763711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.763670    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"]
Oct 30 22:39:58.809523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.809488    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5xmxd"]
Oct 30 22:39:58.878720 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.878687    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"]
Oct 30 22:39:58.879336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.879320    5510 cpu_manager.go:414] "containerMap: removing container" podUID="34a4f229-055c-43b7-a1b9-618a9f575ef2" containerName="revoke-keys"
Oct 30 22:39:58.879395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.879339    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a4f229-055c-43b7-a1b9-618a9f575ef2" containerName="revoke-keys"
Oct 30 22:39:58.879442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.879431    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a4f229-055c-43b7-a1b9-618a9f575ef2" containerName="revoke-keys"
Oct 30 22:39:58.883136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.883114    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:58.931644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.931610    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:58.931804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.931692    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:58.931804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.931733    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:58.931923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.931774    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:39:58.937335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:58.937309    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"]
Oct 30 22:39:59.009609 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.009574    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.009829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.009622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf57g\" (UniqueName: \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.009829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.009772    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.110871 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.110803    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.111006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.110889    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.111006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.110909    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf57g\" (UniqueName: \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.111521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.111490    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.113124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.113097    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.124289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.124266    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355220cd-e662-4c42-aecd-e69f875c507f" path="/var/lib/kubelet/pods/355220cd-e662-4c42-aecd-e69f875c507f/volumes"
Oct 30 22:39:59.163707 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.163680    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf57g\" (UniqueName: \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\") pod \"maas-api-revoke-sub-3d4156f1-d455m\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.198592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.198570    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:39:59.563728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:39:59.563693    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"]
Oct 30 22:39:59.571406 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:39:59.571376    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00897a4_79b9_45ab_84b6_0734d9dd658e.slice/crio-bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b WatchSource:0}: Error finding container bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b: Status 404 returned error can't find the container with id bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b
Oct 30 22:40:00.017199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:00.017167    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m" event={"ID":"e00897a4-79b9-45ab-84b6-0734d9dd658e","Type":"ContainerStarted","Data":"1b45fbc37f602c9bfccc64d94e433e9d9fa13049d5aba2dec064331bc10ee87c"}
Oct 30 22:40:00.017593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:00.017218    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m" event={"ID":"e00897a4-79b9-45ab-84b6-0734d9dd658e","Type":"ContainerStarted","Data":"bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b"}
Oct 30 22:40:00.125548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:00.125463    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m" podStartSLOduration=2.125451201 podStartE2EDuration="2.125451201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:00.123497201 +0000 UTC m=+3409.107404078" watchObservedRunningTime="2026-10-30 22:40:00.125451201 +0000 UTC m=+3409.109358077"
Oct 30 22:40:01.026711 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:01.026671    5510 generic.go:356] "Generic (PLEG): container finished" podID="e00897a4-79b9-45ab-84b6-0734d9dd658e" containerID="1b45fbc37f602c9bfccc64d94e433e9d9fa13049d5aba2dec064331bc10ee87c" exitCode=0
Oct 30 22:40:01.027373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:01.027347    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m" event={"ID":"e00897a4-79b9-45ab-84b6-0734d9dd658e","Type":"ContainerDied","Data":"1b45fbc37f602c9bfccc64d94e433e9d9fa13049d5aba2dec064331bc10ee87c"}
Oct 30 22:40:02.161828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.161805    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:40:02.242421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.242386    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\") pod \"e00897a4-79b9-45ab-84b6-0734d9dd658e\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") "
Oct 30 22:40:02.242615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.242454    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\" (UniqueName: \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\") pod \"e00897a4-79b9-45ab-84b6-0734d9dd658e\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") "
Oct 30 22:40:02.242615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.242483    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\") pod \"e00897a4-79b9-45ab-84b6-0734d9dd658e\" (UID: \"e00897a4-79b9-45ab-84b6-0734d9dd658e\") "
Oct 30 22:40:02.242858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.242826    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca" pod "e00897a4-79b9-45ab-84b6-0734d9dd658e" (UID: "e00897a4-79b9-45ab-84b6-0734d9dd658e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:02.242999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.242975    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:02.244512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.244487    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca" pod "e00897a4-79b9-45ab-84b6-0734d9dd658e" (UID: "e00897a4-79b9-45ab-84b6-0734d9dd658e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:02.244619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.244606    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g" pod "e00897a4-79b9-45ab-84b6-0734d9dd658e" (UID: "e00897a4-79b9-45ab-84b6-0734d9dd658e"). InnerVolumeSpecName "kube-api-access-xf57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:02.344050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.343959    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf57g\" (UniqueName: \"kubernetes.io/projected/e00897a4-79b9-45ab-84b6-0734d9dd658e-kube-api-access-xf57g\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:02.344050 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:02.343992    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00897a4-79b9-45ab-84b6-0734d9dd658e-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:03.038087 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:03.038054    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"
Oct 30 22:40:03.038300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:03.038091    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m" event={"ID":"e00897a4-79b9-45ab-84b6-0734d9dd658e","Type":"ContainerDied","Data":"bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b"}
Oct 30 22:40:03.038300 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:03.038139    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed65bc264a9f27780caa65568de12301e1adbe23f25a1611a4e73851ba8216b"
Oct 30 22:40:13.279414 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:40:13.279368    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:40:18.043679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:18.043644    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"]
Oct 30 22:40:18.089765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:18.089725    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vx422"]
Oct 30 22:40:19.123196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:19.123165    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4801a85-d793-49a8-b085-a53060e377a4" path="/var/lib/kubelet/pods/c4801a85-d793-49a8-b085-a53060e377a4/volumes"
Oct 30 22:40:24.934218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:24.934123    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:25.497723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.497689    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:40:25.498139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.498123    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e00897a4-79b9-45ab-84b6-0734d9dd658e" containerName="revoke-keys"
Oct 30 22:40:25.498139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.498139    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00897a4-79b9-45ab-84b6-0734d9dd658e" containerName="revoke-keys"
Oct 30 22:40:25.498282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.498212    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00897a4-79b9-45ab-84b6-0734d9dd658e" containerName="revoke-keys"
Oct 30 22:40:25.502494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.502471    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554541 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554506    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554550    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554667    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554729    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554826    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnnk\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.554924 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554878    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.555005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.554984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.555074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.555060    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.555122 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.555095    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.571253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.571220    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:40:25.583207 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.583180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:25.602598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.602563    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-673c84-openshift-default-dockercfg-pvpnh\""
Oct 30 22:40:25.656441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656412    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656453    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656502    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656533    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656562    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnnk\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656754    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656846    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.656884 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656873    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656941    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.656992    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657200 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.657182    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657293 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.657266    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.657350    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.657427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.657410    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.659384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.659363    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.659702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.659680    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.720565 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.720301    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnnk\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.728893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.728856    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\") pod \"e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") " pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:25.745938 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.745897    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:25.817496 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:25.817409    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:26.238685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:26.238654    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:40:26.245591 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:40:26.245566    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a612103_d46a_4c74_ab15_48268b72f473.slice/crio-7a8f23321be2a8a9ee932c4bcaef0c8c7bd64e4ee8930d15b35206539425a326 WatchSource:0}: Error finding container 7a8f23321be2a8a9ee932c4bcaef0c8c7bd64e4ee8930d15b35206539425a326: Status 404 returned error can't find the container with id 7a8f23321be2a8a9ee932c4bcaef0c8c7bd64e4ee8930d15b35206539425a326
Oct 30 22:40:26.247657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:26.247624    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:26.247761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:26.247690    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:26.247761 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:26.247725    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:27.158530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:27.158491    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerStarted","Data":"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"}
Oct 30 22:40:27.158739 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:27.158542    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerStarted","Data":"7a8f23321be2a8a9ee932c4bcaef0c8c7bd64e4ee8930d15b35206539425a326"}
Oct 30 22:40:27.317325 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:27.317273    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podStartSLOduration=2.3172561959999998 podStartE2EDuration="2.317256196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:27.31494936 +0000 UTC m=+3436.298856238" watchObservedRunningTime="2026-10-30 22:40:27.317256196 +0000 UTC m=+3436.301163116"
Oct 30 22:40:27.817719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:27.817678    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:27.822590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:27.822565    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:28.164527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:28.164438    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:28.165418 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:28.165395    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:29.044940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:29.044903    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"]
Oct 30 22:40:29.089006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:29.088964    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4zz7f"]
Oct 30 22:40:29.123394 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:29.123363    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9312669c-09a7-489a-9c13-b1c51bf78237" path="/var/lib/kubelet/pods/9312669c-09a7-489a-9c13-b1c51bf78237/volumes"
Oct 30 22:40:30.222855 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:30.222818    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:30.420914 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:30.420878    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:30.573645 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:30.573571    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:33.383988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:33.383950    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:33.555254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:33.555224    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.106256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.106224    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:40:36.111112 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.111093    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.159374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.159346    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.159500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.159390    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.159500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.159487    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzh8t\" (UniqueName: \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.159582 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.159530    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.206096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.206061    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-673c84-serving-cert\""
Oct 30 22:40:36.206286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.206106    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 22:40:36.206286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.206123    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:40:36.206286 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.206157    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:40:36.208046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.208024    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:40:36.208748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.208736    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:40:36.260611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.260580    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.260812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.260630    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.260812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.260745    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzh8t\" (UniqueName: \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.260958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.260884    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.260958 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:40:36.260953    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-serving-cert: secret "maas-api-e2e-worker-single-main-673c84-serving-cert" not found
Oct 30 22:40:36.261079 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:40:36.261008    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls podName:e8be5b4c-c8b6-4902-ab97-5cee7d017632 nodeName:}" failed. No retries permitted until 2026-10-30 22:40:36.760988705 +0000 UTC m=+3445.744895576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls") pod "maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" (UID: "e8be5b4c-c8b6-4902-ab97-5cee7d017632") : secret "maas-api-e2e-worker-single-main-673c84-serving-cert" not found
Oct 30 22:40:36.261362 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.261342    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.263442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.263423    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.316089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.316067    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzh8t\" (UniqueName: \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.324337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.324314    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.498644 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.498606    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.675892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.675847    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:36.766204 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.766125    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:36.768473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:36.768451    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:37.024017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:37.023937    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:37.392709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:37.392630    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:40:37.399675 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:40:37.399650    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8be5b4c_c8b6_4902_ab97_5cee7d017632.slice/crio-f372dd10faa072f2af768971629dc171cbedccb2fc57c56511b321f8719a7d7f WatchSource:0}: Error finding container f372dd10faa072f2af768971629dc171cbedccb2fc57c56511b321f8719a7d7f: Status 404 returned error can't find the container with id f372dd10faa072f2af768971629dc171cbedccb2fc57c56511b321f8719a7d7f
Oct 30 22:40:37.401084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:37.401065    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:40:38.215110 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:38.215073    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" event={"ID":"e8be5b4c-c8b6-4902-ab97-5cee7d017632","Type":"ContainerStarted","Data":"f372dd10faa072f2af768971629dc171cbedccb2fc57c56511b321f8719a7d7f"}
Oct 30 22:40:39.222946 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:39.222898    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" event={"ID":"e8be5b4c-c8b6-4902-ab97-5cee7d017632","Type":"ContainerStarted","Data":"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"}
Oct 30 22:40:39.223381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:39.223103    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:39.331629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:39.331575    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" podStartSLOduration=2.442972126 podStartE2EDuration="3.331561424s" totalImagesPullingTime="888.589298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:39.330272393 +0000 UTC m=+3448.314179271" watchObservedRunningTime="2026-10-30 22:40:39.331561424 +0000 UTC m=+3448.315468299"
Oct 30 22:40:40.958979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:40.957906    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:43.278444 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:40:43.278408    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:40:44.167061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:44.167028    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:40:45.233427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:45.233389    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:40:45.578650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:45.578560    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:50.088218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:50.088180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:54.325476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.325421    5510 generic.go:356] "Generic (PLEG): container finished" podID="0a612103-d46a-4c74-ab15-48268b72f473" containerID="e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046" exitCode=0
Oct 30 22:40:54.326294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.326264    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerDied","Data":"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"}
Oct 30 22:40:54.326898 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.326875    5510 scope.go:122] "RemoveContainer" containerID="e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"
Oct 30 22:40:54.331263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.329186    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:54.331263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.329340    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:54.331263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.329439    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:40:54.974803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:54.965435    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:55.333763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:55.333672    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerStarted","Data":"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"}
Oct 30 22:40:55.818312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:55.818278    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:55.819167 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:55.819140    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:55.819292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:55.819192    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:56.818093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:56.818055    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:56.818487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:56.818111    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:57.818513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:57.818477    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:57.819037 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:57.818534    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:58.169858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:58.167882    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:40:58.818865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:58.818823    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:58.819265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:58.818896    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:40:59.480605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:59.480572    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:59.818748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:59.818663    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 22:40:59.818748 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:40:59.818732    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 22:41:00.822046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:00.822003    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:41:00.822991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:00.822967    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:41:04.048056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:04.048019    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"]
Oct 30 22:41:04.095087 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:04.095049    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nvp25"]
Oct 30 22:41:04.210224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:04.210182    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.123319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:05.123251    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102303ef-d5c5-4ca4-8f82-dfea7ce75496" path="/var/lib/kubelet/pods/102303ef-d5c5-4ca4-8f82-dfea7ce75496/volumes"
Oct 30 22:41:05.160330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:05.160298    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.764173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:05.764138    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.935508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:05.935475    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:08.677949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:08.677915    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.162846 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:13.162816    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.279267 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:41:13.279226    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:41:15.937049 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:15.937013    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:17.651041 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:17.650977    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:19.545969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:19.545926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:22.269077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:22.269050    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:23.756874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:23.756835    5510 scope.go:122] "RemoveContainer" containerID="80b586175406b74f434e8fe6d1902bd21b325257aeb8ebb79f3b56e6f9914de3"
Oct 30 22:41:23.765366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:23.765343    5510 scope.go:122] "RemoveContainer" containerID="db2c56ed805af893d5a3003585cd41453e8f3c30cb4a3b6e85b15dfdff77ba1d"
Oct 30 22:41:23.773603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:23.773577    5510 scope.go:122] "RemoveContainer" containerID="becf9ab02199335dfd480a51b157cc46c8df8a806225f3f6845a2dc1f7c15dde"
Oct 30 22:41:24.133426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:24.133341    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:41:26.757376 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:26.757345    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:31.368409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:31.368366    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:35.853381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:35.853345    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:36.044932 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:36.044889    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"]
Oct 30 22:41:36.090962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:36.090932    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bhclr"]
Oct 30 22:41:36.717436 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:36.717401    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:37.123420 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:37.123342    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159a4f65-272e-4f41-9a10-0f8d8970082b" path="/var/lib/kubelet/pods/159a4f65-272e-4f41-9a10-0f8d8970082b/volumes"
Oct 30 22:41:40.359357 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:40.359326    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:43.278815 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:41:43.278756    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:41:44.866728 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:44.866696    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:49.270419 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:49.270366    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:41:49.367423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:49.367390    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:53.862954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:53.862879    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:55.520949 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:55.520916    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.424154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:41:58.422026    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:02.859141 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:02.859098    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:07.043984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:07.043946    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"]
Oct 30 22:42:07.089857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:07.089819    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-57j92"]
Oct 30 22:42:07.123449 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:07.123413    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3152c2e-e002-42ad-9c2d-14d281730df4" path="/var/lib/kubelet/pods/b3152c2e-e002-42ad-9c2d-14d281730df4/volumes"
Oct 30 22:42:07.356145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:07.356072    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:11.853287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:11.853245    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:13.279464 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:42:13.279433    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:42:15.343970 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:15.343933    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:16.487280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:16.487244    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:18.837109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:18.837072    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:42:20.390303 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:20.390263    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:21.377336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:21.377303    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:23.803450 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:23.803418    5510 scope.go:122] "RemoveContainer" containerID="f154ee06d881dd9f4f812ea2e7df96f75278e46ad8e4073e436a34c7e435e67a"
Oct 30 22:42:23.811891 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:23.811871    5510 scope.go:122] "RemoveContainer" containerID="bc0121ae082dec0abfb20a233c368df7f5ee5abc09eb9bc0caf031112619347b"
Oct 30 22:42:25.843174 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:25.843138    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:27.777089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:27.777059    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:27.926192 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:27.926148    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:30.042960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:30.042922    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"]
Oct 30 22:42:30.089023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:30.088992    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7vmps"]
Oct 30 22:42:30.241930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:30.241895    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:31.123650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:31.123616    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0394ed3a-8ee7-446a-9f48-48c61d2ea36a" path="/var/lib/kubelet/pods/0394ed3a-8ee7-446a-9f48-48c61d2ea36a/volumes"
Oct 30 22:42:34.127717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:34.127681    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:42:34.128161 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:34.127907    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerName="maas-api" containerID="cri-o://a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720" gracePeriod=30
Oct 30 22:42:34.243292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:34.243263    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:35.228611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.228575    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.232:8443/health\": dial tcp 10.133.0.232:8443: connect: connection refused"
Oct 30 22:42:35.391570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.391536    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:42:35.455179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455091    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\") pod \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") "
Oct 30 22:42:35.455336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455196    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") pod \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") "
Oct 30 22:42:35.455336 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455328    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\" (UniqueName: \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\") pod \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") "
Oct 30 22:42:35.455439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455358    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\") pod \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\" (UID: \"e8be5b4c-c8b6-4902-ab97-5cee7d017632\") "
Oct 30 22:42:35.455531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455501    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca" pod "e8be5b4c-c8b6-4902-ab97-5cee7d017632" (UID: "e8be5b4c-c8b6-4902-ab97-5cee7d017632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:35.455668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.455650    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8be5b4c-c8b6-4902-ab97-5cee7d017632-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.457424 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.457398    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls" pod "e8be5b4c-c8b6-4902-ab97-5cee7d017632" (UID: "e8be5b4c-c8b6-4902-ab97-5cee7d017632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:35.457523 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.457417    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t" pod "e8be5b4c-c8b6-4902-ab97-5cee7d017632" (UID: "e8be5b4c-c8b6-4902-ab97-5cee7d017632"). InnerVolumeSpecName "kube-api-access-vzh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:35.457811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.457771    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs" pod "e8be5b4c-c8b6-4902-ab97-5cee7d017632" (UID: "e8be5b4c-c8b6-4902-ab97-5cee7d017632"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:35.556907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.556870    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.556907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.556910    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzh8t\" (UniqueName: \"kubernetes.io/projected/e8be5b4c-c8b6-4902-ab97-5cee7d017632-kube-api-access-vzh8t\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.557109 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.556927    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8be5b4c-c8b6-4902-ab97-5cee7d017632-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:35.735979 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.735946    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:35.843587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.843540    5510 generic.go:356] "Generic (PLEG): container finished" podID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerID="a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720" exitCode=0
Oct 30 22:42:35.843587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.843586    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"
Oct 30 22:42:35.843852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.843611    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" event={"ID":"e8be5b4c-c8b6-4902-ab97-5cee7d017632","Type":"ContainerDied","Data":"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"}
Oct 30 22:42:35.843852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.843689    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6" event={"ID":"e8be5b4c-c8b6-4902-ab97-5cee7d017632","Type":"ContainerDied","Data":"f372dd10faa072f2af768971629dc171cbedccb2fc57c56511b321f8719a7d7f"}
Oct 30 22:42:35.843852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.843717    5510 scope.go:122] "RemoveContainer" containerID="a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"
Oct 30 22:42:35.853315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.853288    5510 scope.go:122] "RemoveContainer" containerID="a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"
Oct 30 22:42:35.853571 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:42:35.853551    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720\": container with ID starting with a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720 not found: ID does not exist" containerID="a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"
Oct 30 22:42:35.853619 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.853582    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720"} err="failed to get container status \"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720\": rpc error: code = NotFound desc = could not find container \"a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720\": container with ID starting with a8ea87154d909e11ec710e51f10d648729e96c1b1784b6e118e9d5cfb4238720 not found: ID does not exist"
Oct 30 22:42:35.995598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.995514    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:42:35.999590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:35.999559    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-673c84-664cc6b99f-47hh6"]
Oct 30 22:42:36.005616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:36.005591    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:36.766588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:36.766557    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:37.124522 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:37.124445    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" path="/var/lib/kubelet/pods/e8be5b4c-c8b6-4902-ab97-5cee7d017632/volumes"
Oct 30 22:42:38.436320 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:38.436288    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:39.129478 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:39.129444    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:42:43.072364 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:43.072327    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:43.254704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:43.254674    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:43.279600 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:42:43.279568    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:42:43.429237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:43.429156    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:42:43.429677 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:43.429640    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" containerID="cri-o://615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d" gracePeriod=30
Oct 30 22:42:43.476434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:43.476391    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:45.825431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:45.825390    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:45.825853 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:45.825480    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:47.407622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:47.407588    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:48.370921 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.370888    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:48.691723 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.691693    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:42:48.791972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.791938    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.791997    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792062    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792083    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792144    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792279    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792317    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792329    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:48.792525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792347    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792364    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:48.792525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792433    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\") pod \"0a612103-d46a-4c74-ab15-48268b72f473\" (UID: \"0a612103-d46a-4c74-ab15-48268b72f473\") "
Oct 30 22:42:48.792525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792479    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:48.792819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792600    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:48.792819 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792746    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:48.792940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792886    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a612103-d46a-4c74-ab15-48268b72f473-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.792940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792905    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.792940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792922    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.792940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792937    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.793179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.792952    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.794430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.794397    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:48.794430 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.794413    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "kube-api-access-ppnnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:48.794581 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.794466    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:48.794733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.794717    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo" pod "0a612103-d46a-4c74-ab15-48268b72f473" (UID: "0a612103-d46a-4c74-ab15-48268b72f473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:48.894310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.894273    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnnk\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-kube-api-access-ppnnk\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.894310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.894305    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a612103-d46a-4c74-ab15-48268b72f473-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.894310 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.894315    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a612103-d46a-4c74-ab15-48268b72f473-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.894560 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.894323    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a612103-d46a-4c74-ab15-48268b72f473-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:48.927499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.927421    5510 generic.go:356] "Generic (PLEG): container finished" podID="0a612103-d46a-4c74-ab15-48268b72f473" containerID="615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d" exitCode=0
Oct 30 22:42:48.927499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.927470    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"
Oct 30 22:42:48.927686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.927501    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerDied","Data":"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"}
Oct 30 22:42:48.927686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.927558    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr" event={"ID":"0a612103-d46a-4c74-ab15-48268b72f473","Type":"ContainerDied","Data":"7a8f23321be2a8a9ee932c4bcaef0c8c7bd64e4ee8930d15b35206539425a326"}
Oct 30 22:42:48.927686 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.927610    5510 scope.go:122] "RemoveContainer" containerID="615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"
Oct 30 22:42:48.937111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.937095    5510 scope.go:122] "RemoveContainer" containerID="e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"
Oct 30 22:42:48.944826 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.944804    5510 scope.go:122] "RemoveContainer" containerID="615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"
Oct 30 22:42:48.945060 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:42:48.945044    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d\": container with ID starting with 615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d not found: ID does not exist" containerID="615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"
Oct 30 22:42:48.945118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.945070    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d"} err="failed to get container status \"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d\": rpc error: code = NotFound desc = could not find container \"615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d\": container with ID starting with 615b4be5ccc4a8f6d29923fcf3008d72b81632d87ce2f2e7f09d2eb417ce5d7d not found: ID does not exist"
Oct 30 22:42:48.945118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.945089    5510 scope.go:122] "RemoveContainer" containerID="e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"
Oct 30 22:42:48.945276 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:42:48.945261    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046\": container with ID starting with e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046 not found: ID does not exist" containerID="e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"
Oct 30 22:42:48.945315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:48.945280    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046"} err="failed to get container status \"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046\": rpc error: code = NotFound desc = could not find container \"e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046\": container with ID starting with e1993b3ccdf95507ef2d3f506a5169bd7a98f2c1892bf9fe729044e858506046 not found: ID does not exist"
Oct 30 22:42:49.082812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:49.082759    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:42:49.087269 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:49.087242    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-673c84-openshift-default-8579b87bd995zxr"]
Oct 30 22:42:49.130152 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:49.123959    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a612103-d46a-4c74-ab15-48268b72f473" path="/var/lib/kubelet/pods/0a612103-d46a-4c74-ab15-48268b72f473/volumes"
Oct 30 22:42:50.583116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:50.583083    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:54.415760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:54.415677    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:55.040589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.040552    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"]
Oct 30 22:42:55.086736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.086705    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d455m"]
Oct 30 22:42:55.122673 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.122632    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00897a4-79b9-45ab-84b6-0734d9dd658e" path="/var/lib/kubelet/pods/e00897a4-79b9-45ab-84b6-0734d9dd658e/volumes"
Oct 30 22:42:55.152413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152383    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"]
Oct 30 22:42:55.152799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152772    5510 cpu_manager.go:414] "containerMap: removing container" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerName="maas-api"
Oct 30 22:42:55.152799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152798    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerName="maas-api"
Oct 30 22:42:55.152881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152809    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.152881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152814    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.152881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152828    5510 cpu_manager.go:414] "containerMap: removing container" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.152881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152834    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.153008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152903    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8be5b4c-c8b6-4902-ab97-5cee7d017632" containerName="maas-api"
Oct 30 22:42:55.153008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152911    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.153008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.152919    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a612103-d46a-4c74-ab15-48268b72f473" containerName="istio-proxy"
Oct 30 22:42:55.157509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.157489    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.207428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.207371    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:55.207428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.207407    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:42:55.207428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.207422    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:55.207428 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.207408    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:55.209254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.209230    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"]
Oct 30 22:42:55.257388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.257352    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45txm\" (UniqueName: \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.257559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.257476    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.257559 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.257535    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.359281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.359201    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45txm\" (UniqueName: \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.359281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.359247    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.359281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.359272    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.359966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.359939    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.361592 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.361572    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.412328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.412292    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45txm\" (UniqueName: \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\") pod \"maas-api-revoke-sub-3d4156f1-bp92q\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.469173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.469140    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:55.846922 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.846883    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"]
Oct 30 22:42:55.855072 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:42:55.855043    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a279d03_a238_445d_b073_a53a6f480279.slice/crio-d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088 WatchSource:0}: Error finding container d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088: Status 404 returned error can't find the container with id d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088
Oct 30 22:42:55.964524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:55.964492    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q" event={"ID":"6a279d03-a238-445d-b073-a53a6f480279","Type":"ContainerStarted","Data":"d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088"}
Oct 30 22:42:56.970712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:56.970677    5510 generic.go:356] "Generic (PLEG): container finished" podID="6a279d03-a238-445d-b073-a53a6f480279" containerID="2ff98ce977af01f990a5fe5d995a7ed93068adb8d1509ea855a8236c3530fdf4" exitCode=0
Oct 30 22:42:56.971131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:56.970759    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q" event={"ID":"6a279d03-a238-445d-b073-a53a6f480279","Type":"ContainerDied","Data":"2ff98ce977af01f990a5fe5d995a7ed93068adb8d1509ea855a8236c3530fdf4"}
Oct 30 22:42:58.110601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.110577    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:58.284993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.284914    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\" (UniqueName: \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\") pod \"6a279d03-a238-445d-b073-a53a6f480279\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") "
Oct 30 22:42:58.285151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.285016    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\") pod \"6a279d03-a238-445d-b073-a53a6f480279\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") "
Oct 30 22:42:58.285151 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.285069    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\") pod \"6a279d03-a238-445d-b073-a53a6f480279\" (UID: \"6a279d03-a238-445d-b073-a53a6f480279\") "
Oct 30 22:42:58.285459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.285427    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca" pod "6a279d03-a238-445d-b073-a53a6f480279" (UID: "6a279d03-a238-445d-b073-a53a6f480279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:58.287059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.287037    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm" pod "6a279d03-a238-445d-b073-a53a6f480279" (UID: "6a279d03-a238-445d-b073-a53a6f480279"). InnerVolumeSpecName "kube-api-access-45txm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:58.287163 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.287083    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca" pod "6a279d03-a238-445d-b073-a53a6f480279" (UID: "6a279d03-a238-445d-b073-a53a6f480279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:58.386801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.386742    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a279d03-a238-445d-b073-a53a6f480279-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:58.386978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.386775    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45txm\" (UniqueName: \"kubernetes.io/projected/6a279d03-a238-445d-b073-a53a6f480279-kube-api-access-45txm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:58.386978 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.386842    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a279d03-a238-445d-b073-a53a6f480279-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:58.981160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.981130    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"
Oct 30 22:42:58.981341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.981170    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q" event={"ID":"6a279d03-a238-445d-b073-a53a6f480279","Type":"ContainerDied","Data":"d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088"}
Oct 30 22:42:58.981341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:42:58.981216    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a1ce96292aa9117c24ac58453494b9d243dbf05284e244419684eb7f003088"
Oct 30 22:43:06.992925 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:06.992888    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:07.665506 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:07.665468    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:11.985372 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:11.985343    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:43:12.421623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:12.421551    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:43:13.278423 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:43:13.278384    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:43:13.897063 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:13.897022    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 22:43:13.897315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:13.897287    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" podUID="7f484336-ce43-403b-9024-25f71d813905" containerName="manager" containerID="cri-o://460c2a27231138acdee0e6dacdeec96da2dc7a61b02b5e53b3ee2d18cef2135a" gracePeriod=10
Oct 30 22:43:14.062491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.062451    5510 generic.go:356] "Generic (PLEG): container finished" podID="7f484336-ce43-403b-9024-25f71d813905" containerID="460c2a27231138acdee0e6dacdeec96da2dc7a61b02b5e53b3ee2d18cef2135a" exitCode=0
Oct 30 22:43:14.062687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.062520    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" event={"ID":"7f484336-ce43-403b-9024-25f71d813905","Type":"ContainerDied","Data":"460c2a27231138acdee0e6dacdeec96da2dc7a61b02b5e53b3ee2d18cef2135a"}
Oct 30 22:43:14.147128 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.147070    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 22:43:14.217903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.217875    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\") pod \"7f484336-ce43-403b-9024-25f71d813905\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") "
Oct 30 22:43:14.218104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.217977    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\" (UniqueName: \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\") pod \"7f484336-ce43-403b-9024-25f71d813905\" (UID: \"7f484336-ce43-403b-9024-25f71d813905\") "
Oct 30 22:43:14.218238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.218215    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume" pod "7f484336-ce43-403b-9024-25f71d813905" (UID: "7f484336-ce43-403b-9024-25f71d813905"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:14.220123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.220101    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2" pod "7f484336-ce43-403b-9024-25f71d813905" (UID: "7f484336-ce43-403b-9024-25f71d813905"). InnerVolumeSpecName "kube-api-access-khkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:14.319032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.318993    5510 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f484336-ce43-403b-9024-25f71d813905-extensions-socket-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:14.319032 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:14.319025    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkh2\" (UniqueName: \"kubernetes.io/projected/7f484336-ce43-403b-9024-25f71d813905-kube-api-access-khkh2\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:15.069370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:15.069338    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"
Oct 30 22:43:15.069583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:15.069375    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj" event={"ID":"7f484336-ce43-403b-9024-25f71d813905","Type":"ContainerDied","Data":"f0ee5db95f3947c4740eee05ec4857fc7dc80f659d244affc9d90cbcc5c87e43"}
Oct 30 22:43:15.069583 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:15.069446    5510 scope.go:122] "RemoveContainer" containerID="460c2a27231138acdee0e6dacdeec96da2dc7a61b02b5e53b3ee2d18cef2135a"
Oct 30 22:43:15.269993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:15.269950    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 22:43:15.316023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:15.315989    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fx5hj"]
Oct 30 22:43:17.123129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:17.123100    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f484336-ce43-403b-9024-25f71d813905" path="/var/lib/kubelet/pods/7f484336-ce43-403b-9024-25f71d813905/volumes"
Oct 30 22:43:23.834355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:23.834274    5510 scope.go:122] "RemoveContainer" containerID="5a03d7b1397afabf6d7c3d21f9046a7af6984125ba7085c40a3a4219c4890303"
Oct 30 22:43:23.842323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:23.842290    5510 scope.go:122] "RemoveContainer" containerID="be695aa2201890b9e5bca0e7d83ce8ca2a6c0e32be1188aa288947bfb531f8c0"
Oct 30 22:43:37.432845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:43:37.432808    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:43.278560 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:43:43.278524    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:44:13.278588 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:44:13.278546    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:44:17.572486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572448    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"]
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572845    5510 cpu_manager.go:414] "containerMap: removing container" podUID="7f484336-ce43-403b-9024-25f71d813905" containerName="manager"
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572860    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f484336-ce43-403b-9024-25f71d813905" containerName="manager"
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572884    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6a279d03-a238-445d-b073-a53a6f480279" containerName="revoke-keys"
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572889    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a279d03-a238-445d-b073-a53a6f480279" containerName="revoke-keys"
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572961    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f484336-ce43-403b-9024-25f71d813905" containerName="manager"
Oct 30 22:44:17.573051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.572975    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a279d03-a238-445d-b073-a53a6f480279" containerName="revoke-keys"
Oct 30 22:44:17.575920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.575904    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.625927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.625885    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fxvwt\""
Oct 30 22:44:17.679683 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.674074    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"]
Oct 30 22:44:17.708011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.707975    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkt6\" (UniqueName: \"kubernetes.io/projected/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-kube-api-access-sfkt6\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.708206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.708032    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.809051 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.809007    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkt6\" (UniqueName: \"kubernetes.io/projected/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-kube-api-access-sfkt6\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.809239 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.809063    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.809435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.809418    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.862218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.862146    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkt6\" (UniqueName: \"kubernetes.io/projected/e8987d6e-68b6-48ca-91a9-7d8a706d9b8a-kube-api-access-sfkt6\") pod \"kuadrant-operator-controller-manager-678d87766f-tj846\" (UID: \"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:17.889021 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:17.888993    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:18.360444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:18.360408    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"]
Oct 30 22:44:18.368686 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:44:18.368658    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8987d6e_68b6_48ca_91a9_7d8a706d9b8a.slice/crio-cad9b2d5194cd2997b211a6f760f4cd47c2618e3188bbb3fcf88afe3109c293d WatchSource:0}: Error finding container cad9b2d5194cd2997b211a6f760f4cd47c2618e3188bbb3fcf88afe3109c293d: Status 404 returned error can't find the container with id cad9b2d5194cd2997b211a6f760f4cd47c2618e3188bbb3fcf88afe3109c293d
Oct 30 22:44:19.382062 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:19.382028    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846" event={"ID":"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a","Type":"ContainerStarted","Data":"ef3684fc7200442f13bd842fe8012f8e3371e19f915eb931152ed8e607ea33fb"}
Oct 30 22:44:19.382509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:19.382087    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846" event={"ID":"e8987d6e-68b6-48ca-91a9-7d8a706d9b8a","Type":"ContainerStarted","Data":"cad9b2d5194cd2997b211a6f760f4cd47c2618e3188bbb3fcf88afe3109c293d"}
Oct 30 22:44:19.382509 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:19.382243    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:19.584093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:19.584037    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846" podStartSLOduration=2.584021384 podStartE2EDuration="2.584021384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:19.536904353 +0000 UTC m=+3668.520811222" watchObservedRunningTime="2026-10-30 22:44:19.584021384 +0000 UTC m=+3668.567928259"
Oct 30 22:44:23.876089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:23.876052    5510 scope.go:122] "RemoveContainer" containerID="528e443233394efe0542545e03ca66fca2ff8893a2ca6342705ae279bf46af56"
Oct 30 22:44:23.884546 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:23.884523    5510 scope.go:122] "RemoveContainer" containerID="8f2a4d5164b0415de699a3ec24748abed434d56b410c1d3f3e2038ab969e2f14"
Oct 30 22:44:23.891993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:23.891973    5510 scope.go:122] "RemoveContainer" containerID="a9f0d1f8d07544fd73f21df19abe9682ab6cbb398d7fae33a41ac656a3bf8540"
Oct 30 22:44:30.388632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:30.388602    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj846"
Oct 30 22:44:32.361672 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:32.361616    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:32.570299 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:32.570246    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:34.045330 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:34.045294    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"]
Oct 30 22:44:34.091494 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:34.091461    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jlbw6"]
Oct 30 22:44:35.123396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:35.123359    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96164e39-9322-409b-b9c7-f79f05405da8" path="/var/lib/kubelet/pods/96164e39-9322-409b-b9c7-f79f05405da8/volumes"
Oct 30 22:44:37.697459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:37.697413    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:39.562607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:39.562569    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:44:43.283689 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:44:43.283653    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:44:46.053390 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:46.053347    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"]
Oct 30 22:44:46.099342 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:46.099306    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xrg6l"]
Oct 30 22:44:47.123795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:47.123755    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a4f229-055c-43b7-a1b9-618a9f575ef2" path="/var/lib/kubelet/pods/34a4f229-055c-43b7-a1b9-618a9f575ef2/volumes"
Oct 30 22:44:53.625858 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.625766    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"]
Oct 30 22:44:53.629625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.629605    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.663969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.663919    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.664144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.663980    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.664144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.664119    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwpn\" (UniqueName: \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.682622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.682587    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"]
Oct 30 22:44:53.726879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.726845    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:44:53.726879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.726841    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:53.727113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.726948    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:44:53.727113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.726947    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:44:53.764770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.764734    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.765011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.764804    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.765011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.764871    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwpn\" (UniqueName: \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.765472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.765446    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.767147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.767129    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.819732 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.819696    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwpn\" (UniqueName: \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\") pod \"maas-api-revoke-sub-d576e68b-r5w84\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:53.942539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:53.942458    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:54.310116 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:54.310082    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"]
Oct 30 22:44:54.318332 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:44:54.318300    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916db88_cc98_40d5_aefb_c901b0a1386e.slice/crio-9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550 WatchSource:0}: Error finding container 9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550: Status 404 returned error can't find the container with id 9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550
Oct 30 22:44:54.559527 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:54.559499    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84" event={"ID":"6916db88-cc98-40d5-aefb-c901b0a1386e","Type":"ContainerStarted","Data":"9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550"}
Oct 30 22:44:55.565626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:55.565586    5510 generic.go:356] "Generic (PLEG): container finished" podID="6916db88-cc98-40d5-aefb-c901b0a1386e" containerID="f8b1bf742c807a93ec92f92a04eacb535564b2ccddf73c91932a6861bb5bd925" exitCode=0
Oct 30 22:44:55.566059 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:55.565636    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84" event={"ID":"6916db88-cc98-40d5-aefb-c901b0a1386e","Type":"ContainerDied","Data":"f8b1bf742c807a93ec92f92a04eacb535564b2ccddf73c91932a6861bb5bd925"}
Oct 30 22:44:56.697429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.697400    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:44:56.793306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.793271    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\") pod \"6916db88-cc98-40d5-aefb-c901b0a1386e\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") "
Oct 30 22:44:56.793483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.793355    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\" (UniqueName: \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\") pod \"6916db88-cc98-40d5-aefb-c901b0a1386e\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") "
Oct 30 22:44:56.793483 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.793382    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\") pod \"6916db88-cc98-40d5-aefb-c901b0a1386e\" (UID: \"6916db88-cc98-40d5-aefb-c901b0a1386e\") "
Oct 30 22:44:56.793823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.793794    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca" pod "6916db88-cc98-40d5-aefb-c901b0a1386e" (UID: "6916db88-cc98-40d5-aefb-c901b0a1386e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:56.795327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.795294    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca" pod "6916db88-cc98-40d5-aefb-c901b0a1386e" (UID: "6916db88-cc98-40d5-aefb-c901b0a1386e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:56.795422 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.795355    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn" pod "6916db88-cc98-40d5-aefb-c901b0a1386e" (UID: "6916db88-cc98-40d5-aefb-c901b0a1386e"). InnerVolumeSpecName "kube-api-access-qmwpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:56.894767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.894686    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwpn\" (UniqueName: \"kubernetes.io/projected/6916db88-cc98-40d5-aefb-c901b0a1386e-kube-api-access-qmwpn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:56.894767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.894718    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:56.894767 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:56.894728    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6916db88-cc98-40d5-aefb-c901b0a1386e-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:57.576469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:57.576433    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84" event={"ID":"6916db88-cc98-40d5-aefb-c901b0a1386e","Type":"ContainerDied","Data":"9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550"}
Oct 30 22:44:57.576657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:57.576483    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5c2a9b3a2ea9ad85ee925c99abacf1e51412a120f915c3b94a5e0da75fb550"
Oct 30 22:44:57.576657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:44:57.576449    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"
Oct 30 22:45:00.164245 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.164211    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:00.164638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.164612    5510 cpu_manager.go:414] "containerMap: removing container" podUID="6916db88-cc98-40d5-aefb-c901b0a1386e" containerName="revoke-keys"
Oct 30 22:45:00.164638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.164623    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6916db88-cc98-40d5-aefb-c901b0a1386e" containerName="revoke-keys"
Oct 30 22:45:00.164715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.164692    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6916db88-cc98-40d5-aefb-c901b0a1386e" containerName="revoke-keys"
Oct 30 22:45:00.167849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.167817    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.217025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.216989    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.226638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.226601    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjk8\" (UniqueName: \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.226848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.226655    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.263045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.263009    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:45:00.263045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.263009    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.264625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.264605    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:00.327520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.327484    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjk8\" (UniqueName: \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.327520 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.327527    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.328157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.328138    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.381734 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.381696    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjk8\" (UniqueName: \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\") pod \"maas-api-key-cleanup-29890005-f9g6l\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.486139 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.486099    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:00.853741 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:00.853660    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:00.861692 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:45:00.861657    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b4d189_504c_4268_b1e4_8673fa443832.slice/crio-44239044d652036f8b01885b2e11094d386512b240078d7b787b8687838a240e WatchSource:0}: Error finding container 44239044d652036f8b01885b2e11094d386512b240078d7b787b8687838a240e: Status 404 returned error can't find the container with id 44239044d652036f8b01885b2e11094d386512b240078d7b787b8687838a240e
Oct 30 22:45:01.596879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:01.596838    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerStarted","Data":"614b70dbf58539a22eb4e98a36a393f927aa90ae03b395c9136425297f86eb6e"}
Oct 30 22:45:01.597306 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:01.596906    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerStarted","Data":"44239044d652036f8b01885b2e11094d386512b240078d7b787b8687838a240e"}
Oct 30 22:45:01.703755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:01.703698    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" podStartSLOduration=1.70368277 podStartE2EDuration="1.70368277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.702868698 +0000 UTC m=+3710.686775573" watchObservedRunningTime="2026-10-30 22:45:01.70368277 +0000 UTC m=+3710.687589640"
Oct 30 22:45:02.602625 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:02.602587    5510 generic.go:356] "Generic (PLEG): container finished" podID="56b4d189-504c-4268-b1e4-8673fa443832" containerID="614b70dbf58539a22eb4e98a36a393f927aa90ae03b395c9136425297f86eb6e" exitCode=60
Oct 30 22:45:02.603071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:02.602676    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerDied","Data":"614b70dbf58539a22eb4e98a36a393f927aa90ae03b395c9136425297f86eb6e"}
Oct 30 22:45:02.603071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:02.602898    5510 scope.go:122] "RemoveContainer" containerID="614b70dbf58539a22eb4e98a36a393f927aa90ae03b395c9136425297f86eb6e"
Oct 30 22:45:03.609558 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:03.609468    5510 generic.go:356] "Generic (PLEG): container finished" podID="56b4d189-504c-4268-b1e4-8673fa443832" containerID="796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce" exitCode=60
Oct 30 22:45:03.609996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:03.609550    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerDied","Data":"796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce"}
Oct 30 22:45:03.609996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:03.609613    5510 scope.go:122] "RemoveContainer" containerID="614b70dbf58539a22eb4e98a36a393f927aa90ae03b395c9136425297f86eb6e"
Oct 30 22:45:03.609996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:03.609818    5510 scope.go:122] "RemoveContainer" containerID="796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce"
Oct 30 22:45:03.610139 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:03.610096    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-f9g6l_odh-ai-gateway-infra(56b4d189-504c-4268-b1e4-8673fa443832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" podUID="56b4d189-504c-4268-b1e4-8673fa443832"
Oct 30 22:45:04.615915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:04.615886    5510 scope.go:122] "RemoveContainer" containerID="796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce"
Oct 30 22:45:04.616285 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:04.616128    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-f9g6l_odh-ai-gateway-infra(56b4d189-504c-4268-b1e4-8673fa443832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" podUID="56b4d189-504c-4268-b1e4-8673fa443832"
Oct 30 22:45:06.340260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:06.340220    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:45:13.342220 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:13.342175    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:45:15.673318 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:15.673281    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:45:17.118189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:17.118154    5510 scope.go:122] "RemoveContainer" containerID="796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce"
Oct 30 22:45:17.703373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:17.703340    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerStarted","Data":"b841878770db6345b2c74d9ff6931251b52639ac14cfbd29cb2fba26683218d7"}
Oct 30 22:45:18.709793 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:18.709744    5510 generic.go:356] "Generic (PLEG): container finished" podID="56b4d189-504c-4268-b1e4-8673fa443832" containerID="b841878770db6345b2c74d9ff6931251b52639ac14cfbd29cb2fba26683218d7" exitCode=60
Oct 30 22:45:18.710219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:18.709823    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerDied","Data":"b841878770db6345b2c74d9ff6931251b52639ac14cfbd29cb2fba26683218d7"}
Oct 30 22:45:18.710219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:18.709917    5510 scope.go:122] "RemoveContainer" containerID="796a7125901ec8adccb76dd242beceb19f86f858979da7f0f2838e90ab0eecce"
Oct 30 22:45:18.710219 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:18.710190    5510 scope.go:122] "RemoveContainer" containerID="b841878770db6345b2c74d9ff6931251b52639ac14cfbd29cb2fba26683218d7"
Oct 30 22:45:18.710443 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:18.710425    5510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-f9g6l_odh-ai-gateway-infra(56b4d189-504c-4268-b1e4-8673fa443832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" podUID="56b4d189-504c-4268-b1e4-8673fa443832"
Oct 30 22:45:18.827136 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:18.827101    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:19.853055 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:19.853026    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:19.947379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:19.947334    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\" (UniqueName: \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\") pod \"56b4d189-504c-4268-b1e4-8673fa443832\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") "
Oct 30 22:45:19.947525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:19.947485    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\") pod \"56b4d189-504c-4268-b1e4-8673fa443832\" (UID: \"56b4d189-504c-4268-b1e4-8673fa443832\") "
Oct 30 22:45:19.947870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:19.947848    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca" pod "56b4d189-504c-4268-b1e4-8673fa443832" (UID: "56b4d189-504c-4268-b1e4-8673fa443832"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.949470 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:19.949447    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8" pod "56b4d189-504c-4268-b1e4-8673fa443832" (UID: "56b4d189-504c-4268-b1e4-8673fa443832"). InnerVolumeSpecName "kube-api-access-rvjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.049068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.048992    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4d189-504c-4268-b1e4-8673fa443832-maas-api-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.049068 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.049017    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjk8\" (UniqueName: \"kubernetes.io/projected/56b4d189-504c-4268-b1e4-8673fa443832-kube-api-access-rvjk8\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.721486 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.721457    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"
Oct 30 22:45:20.721712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.721501    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l" event={"ID":"56b4d189-504c-4268-b1e4-8673fa443832","Type":"ContainerDied","Data":"44239044d652036f8b01885b2e11094d386512b240078d7b787b8687838a240e"}
Oct 30 22:45:20.721712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.721557    5510 scope.go:122] "RemoveContainer" containerID="b841878770db6345b2c74d9ff6931251b52639ac14cfbd29cb2fba26683218d7"
Oct 30 22:45:20.873457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.873423    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:20.919115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:20.919084    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-f9g6l"]
Oct 30 22:45:21.124719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:21.124639    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b4d189-504c-4268-b1e4-8673fa443832" path="/var/lib/kubelet/pods/56b4d189-504c-4268-b1e4-8673fa443832/volumes"
Oct 30 22:45:22.581966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.581925    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"]
Oct 30 22:45:22.582368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582335    5510 cpu_manager.go:414] "containerMap: removing container" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582348    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582368 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582367    5510 cpu_manager.go:414] "containerMap: removing container" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582373    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582397    5510 cpu_manager.go:414] "containerMap: removing container" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582402    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582473 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582469    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.582601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.582484    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:22.587037 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.587015    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.637646 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.637612    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:22.637907 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.637636    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:22.683471 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.683430    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:45:22.683637 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.683472    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:22.685047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.685023    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"]
Oct 30 22:45:22.774332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.774294    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.774526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.774434    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.774526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.774496    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklgh\" (UniqueName: \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.875038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.874949    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklgh\" (UniqueName: \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.875038 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.875016    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.875234 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.875080    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.875680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.875649    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.877455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.877431    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:22.928687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:22.928651    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklgh\" (UniqueName: \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\") pod \"maas-api-revoke-sub-d2bc93bc-hgqrn\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:23.199726 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:23.199645    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:23.567429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:23.567394    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"]
Oct 30 22:45:23.575181 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:45:23.575155    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ce9ab9_449a_4823_98b2_d2ea261db9b8.slice/crio-67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44 WatchSource:0}: Error finding container 67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44: Status 404 returned error can't find the container with id 67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44
Oct 30 22:45:23.737246 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:23.737209    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn" event={"ID":"a5ce9ab9-449a-4823-98b2-d2ea261db9b8","Type":"ContainerStarted","Data":"67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44"}
Oct 30 22:45:24.743015 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:24.742981    5510 generic.go:356] "Generic (PLEG): container finished" podID="a5ce9ab9-449a-4823-98b2-d2ea261db9b8" containerID="a4bb1acd30c548d1ffed54bc7607348dbcef3ac5a304c89a29257b4b9724aac1" exitCode=0
Oct 30 22:45:24.743472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:24.743064    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn" event={"ID":"a5ce9ab9-449a-4823-98b2-d2ea261db9b8","Type":"ContainerDied","Data":"a4bb1acd30c548d1ffed54bc7607348dbcef3ac5a304c89a29257b4b9724aac1"}
Oct 30 22:45:25.877183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:25.877158    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:25.899131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:25.899105    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\") pod \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") "
Oct 30 22:45:25.899237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:25.899159    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\" (UniqueName: \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\") pod \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") "
Oct 30 22:45:25.901154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:25.901125    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca" pod "a5ce9ab9-449a-4823-98b2-d2ea261db9b8" (UID: "a5ce9ab9-449a-4823-98b2-d2ea261db9b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:25.901280 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:25.901226    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh" pod "a5ce9ab9-449a-4823-98b2-d2ea261db9b8" (UID: "a5ce9ab9-449a-4823-98b2-d2ea261db9b8"). InnerVolumeSpecName "kube-api-access-qklgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:26.000388 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.000298    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\") pod \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\" (UID: \"a5ce9ab9-449a-4823-98b2-d2ea261db9b8\") "
Oct 30 22:45:26.000675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.000650    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca" pod "a5ce9ab9-449a-4823-98b2-d2ea261db9b8" (UID: "a5ce9ab9-449a-4823-98b2-d2ea261db9b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:26.000735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.000684    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklgh\" (UniqueName: \"kubernetes.io/projected/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-kube-api-access-qklgh\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.000735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.000707    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.101272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.101237    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ce9ab9-449a-4823-98b2-d2ea261db9b8-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:26.754069 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.754032    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn" event={"ID":"a5ce9ab9-449a-4823-98b2-d2ea261db9b8","Type":"ContainerDied","Data":"67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44"}
Oct 30 22:45:26.754251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.754073    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"
Oct 30 22:45:26.754251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:26.754083    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67af69bd9a75b20d9bc24790c375fb4677065aa62e51a0d2bb685684bb523a44"
Oct 30 22:45:35.069178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.069140    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.356447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.356360    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:45:35.632828 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.632728    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:45:35.633179 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.633164    5510 cpu_manager.go:414] "containerMap: removing container" podUID="a5ce9ab9-449a-4823-98b2-d2ea261db9b8" containerName="revoke-keys"
Oct 30 22:45:35.633228 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.633183    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ce9ab9-449a-4823-98b2-d2ea261db9b8" containerName="revoke-keys"
Oct 30 22:45:35.633288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.633279    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b4d189-504c-4268-b1e4-8673fa443832" containerName="cleanup"
Oct 30 22:45:35.633326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.633292    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ce9ab9-449a-4823-98b2-d2ea261db9b8" containerName="revoke-keys"
Oct 30 22:45:35.636892 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.636872    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703240    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703306    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703337    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703379    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703410    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6m8\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703448    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703498    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703527    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.705165 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.703585    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.731829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.731772    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b1b8bb-openshift-default-dockercfg-49kwf\""
Oct 30 22:45:35.733467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.733434    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:45:35.745426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.745380    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.806237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806090    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806192    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806218    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806259    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806287    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6m8\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806324    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806375    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806401    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806456    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.806948 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.806927    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.807530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.807480    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.807737 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.807697    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.807937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.807919    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.808676 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.808651    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.810983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.810962    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.811241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.811223    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.861623 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.861584    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.861830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.861769    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6m8\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\") pod \"e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") " pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:35.922702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.922603    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:35.955488 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:35.955454    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:36.378315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.378282    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:45:36.385750 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:45:36.385727    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44da6002_e234_42da_88c8_fb62db60ba1f.slice/crio-624ac16d94762363fdddf25ea79e366c566a4df6830d8d5623764a7856bbf44d WatchSource:0}: Error finding container 624ac16d94762363fdddf25ea79e366c566a4df6830d8d5623764a7856bbf44d: Status 404 returned error can't find the container with id 624ac16d94762363fdddf25ea79e366c566a4df6830d8d5623764a7856bbf44d
Oct 30 22:45:36.391104 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.391070    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:36.391191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.391155    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:36.391231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.391195    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:45:36.810639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.810601    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerStarted","Data":"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"}
Oct 30 22:45:36.810856 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.810657    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerStarted","Data":"624ac16d94762363fdddf25ea79e366c566a4df6830d8d5623764a7856bbf44d"}
Oct 30 22:45:36.936940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.936886    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podStartSLOduration=1.936871912 podStartE2EDuration="1.936871912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:36.936851123 +0000 UTC m=+3745.920758000" watchObservedRunningTime="2026-10-30 22:45:36.936871912 +0000 UTC m=+3745.920778855"
Oct 30 22:45:36.955940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:36.955908    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:37.961058 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:37.961028    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:38.048795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:38.048751    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:38.820584 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:38.820552    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:38.821700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:38.821673    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:45:39.898604 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:39.898570    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:40.087857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:40.087825    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:40.261533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:40.261487    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.033705 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:43.033675    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.207881 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:43.207847    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:43.320798 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:43.318987    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:45:45.819626 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.819591    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:45:45.823251 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.823228    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:45.872080 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.872048    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b1b8bb-serving-cert\""
Oct 30 22:45:45.906607 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.906573    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:45.906804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.906618    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:45.906804 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.906709    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rdp\" (UniqueName: \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:45.906904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.906810    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:45.918323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.918294    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:45.918323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.918294    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:45.918513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.918363    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:45:45.918513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.918441    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 22:45:45.920504 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:45.920482    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:45:46.008468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.008426    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.008642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.008516    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.008642 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:46.008563    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-serving-cert: secret "maas-api-e2e-worker-single-main-b1b8bb-serving-cert" not found
Oct 30 22:45:46.008642 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.008570    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rdp\" (UniqueName: \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.008642 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:45:46.008638    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls podName:f9d4d72e-f330-4e4c-b764-462f9e95871a nodeName:}" failed. No retries permitted until 2026-10-30 22:45:46.508619934 +0000 UTC m=+3755.492526791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls") pod "maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" (UID: "f9d4d72e-f330-4e4c-b764-462f9e95871a") : secret "maas-api-e2e-worker-single-main-b1b8bb-serving-cert" not found
Oct 30 22:45:46.008912 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.008689    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.009260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.009241    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.011339 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.011310    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.031260 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.031231    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.074421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.074345    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rdp\" (UniqueName: \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.291995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.291962    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.451983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.451885    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.515254 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.515216    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.517598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.517569    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:46.739081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:46.739044    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:47.110920 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:47.110841    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:45:47.117533 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:45:47.117495    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d4d72e_f330_4e4c_b764_462f9e95871a.slice/crio-18812ca395cee64471eec74a52cbc1ebc7ec3c3c493b9faa3939b25eac2c1d5b WatchSource:0}: Error finding container 18812ca395cee64471eec74a52cbc1ebc7ec3c3c493b9faa3939b25eac2c1d5b: Status 404 returned error can't find the container with id 18812ca395cee64471eec74a52cbc1ebc7ec3c3c493b9faa3939b25eac2c1d5b
Oct 30 22:45:47.119047 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:47.119029    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:45:47.871140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:47.871091    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" event={"ID":"f9d4d72e-f330-4e4c-b764-462f9e95871a","Type":"ContainerStarted","Data":"18812ca395cee64471eec74a52cbc1ebc7ec3c3c493b9faa3939b25eac2c1d5b"}
Oct 30 22:45:48.878615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:48.878579    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" event={"ID":"f9d4d72e-f330-4e4c-b764-462f9e95871a","Type":"ContainerStarted","Data":"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"}
Oct 30 22:45:48.879054 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:48.878646    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:48.989315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:48.989264    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" podStartSLOduration=2.956580538 podStartE2EDuration="3.989249368s" totalImagesPullingTime="1.03266883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:48.988315028 +0000 UTC m=+3757.972221909" watchObservedRunningTime="2026-10-30 22:45:48.989249368 +0000 UTC m=+3757.973156244"
Oct 30 22:45:50.664288 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:50.664250    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:53.763256 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:53.763206    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:45:54.889072 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:54.889046    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:45:55.311347 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:55.311316    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:59.986014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:45:59.985963    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:02.947377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.947285    5510 generic.go:356] "Generic (PLEG): container finished" podID="44da6002-e234-42da-88c8-fb62db60ba1f" containerID="4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e" exitCode=0
Oct 30 22:46:02.947377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.947359    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerDied","Data":"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"}
Oct 30 22:46:02.947866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.947846    5510 scope.go:122] "RemoveContainer" containerID="4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"
Oct 30 22:46:02.949045 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.949017    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:02.949164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.949084    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:02.949164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:02.949116    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:46:03.978874 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:03.978832    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerStarted","Data":"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"}
Oct 30 22:46:04.779614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:04.779574    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:04.971315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:04.971280    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:46:04.975802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:04.974053    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:46:04.975802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:04.974112    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 22:46:04.977679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:04.977633    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.407927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:05.407876    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.582746 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:05.582691    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.777429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:05.777395    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.970807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:05.956154    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:46:05.970807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:05.956217    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 22:46:06.964807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:06.961182    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:46:06.964807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:06.961247    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 22:46:07.963806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:07.956189    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:46:07.963806 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:07.956255    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 22:46:08.630718 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:08.630686    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:08.820893 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:08.820849    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:46:08.956669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:08.956571    5510 patch_prober.go:29] interesting pod/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 22:46:08.956669 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:08.956634    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 22:46:09.665533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:09.665498    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:09.960485 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:09.960459    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:46:09.961618 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:09.961594    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:46:13.153323 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:13.153279    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:46:13.282171 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:46:13.282137    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:46:14.265656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:14.265622    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:18.870328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:18.870281    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:23.372475 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:23.372394    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:23.932954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:23.932924    5510 scope.go:122] "RemoveContainer" containerID="1b45fbc37f602c9bfccc64d94e433e9d9fa13049d5aba2dec064331bc10ee87c"
Oct 30 22:46:23.941439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:23.941413    5510 scope.go:122] "RemoveContainer" containerID="1063a6864fac8388e748493290e07f1e6146eb886972da84a79f8d2c6df0eaba"
Oct 30 22:46:23.949479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:23.949456    5510 scope.go:122] "RemoveContainer" containerID="843945958d8dfb7b6dde23d8ea4d6d3bf717504a9780cb91434dc2965549e093"
Oct 30 22:46:27.856612 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:27.856577    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:32.368274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:32.368240    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:36.883597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:36.883561    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:38.359328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:38.359285    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:38.674936 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:38.674842    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:46:41.466284 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:41.466246    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.279606 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:46:43.279560    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:46:46.047795 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:46.047748    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:50.648081 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:50.648046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:55.211867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:55.211807    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:59.658638 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:46:59.658608    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.091976 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:04.091910    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:04.776498 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:04.776466    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:06.627941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:06.627904    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:47:08.885162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:08.885125    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:09.822733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:09.822686    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:13.278821 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:47:13.278765    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:47:13.457434 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:13.457404    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:16.981012 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:16.980961    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:17.161252 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.161180    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:17.287146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.287067    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:47:17.345991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.345948    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" podUID="f9d4d72e-f330-4e4c-b764-462f9e95871a" containerName="maas-api" containerID="cri-o://7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6" gracePeriod=30
Oct 30 22:47:17.411744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.411715    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:17.604561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.604534    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:47:17.619861 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.619831    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\") pod \"f9d4d72e-f330-4e4c-b764-462f9e95871a\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") "
Oct 30 22:47:17.619960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.619904    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\") pod \"f9d4d72e-f330-4e4c-b764-462f9e95871a\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") "
Oct 30 22:47:17.619960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.619949    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") pod \"f9d4d72e-f330-4e4c-b764-462f9e95871a\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") "
Oct 30 22:47:17.620039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.619981    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\" (UniqueName: \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\") pod \"f9d4d72e-f330-4e4c-b764-462f9e95871a\" (UID: \"f9d4d72e-f330-4e4c-b764-462f9e95871a\") "
Oct 30 22:47:17.620327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.620299    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca" pod "f9d4d72e-f330-4e4c-b764-462f9e95871a" (UID: "f9d4d72e-f330-4e4c-b764-462f9e95871a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:17.621956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.621925    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs" pod "f9d4d72e-f330-4e4c-b764-462f9e95871a" (UID: "f9d4d72e-f330-4e4c-b764-462f9e95871a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:17.622106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.622029    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls" pod "f9d4d72e-f330-4e4c-b764-462f9e95871a" (UID: "f9d4d72e-f330-4e4c-b764-462f9e95871a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:17.622106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.622053    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp" pod "f9d4d72e-f330-4e4c-b764-462f9e95871a" (UID: "f9d4d72e-f330-4e4c-b764-462f9e95871a"). InnerVolumeSpecName "kube-api-access-76rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:17.721717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.721675    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d4d72e-f330-4e4c-b764-462f9e95871a-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:17.721717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.721714    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:17.721956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.721728    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rdp\" (UniqueName: \"kubernetes.io/projected/f9d4d72e-f330-4e4c-b764-462f9e95871a-kube-api-access-76rdp\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:17.721956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:17.721737    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9d4d72e-f330-4e4c-b764-462f9e95871a-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:18.351526 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.351493    5510 generic.go:356] "Generic (PLEG): container finished" podID="f9d4d72e-f330-4e4c-b764-462f9e95871a" containerID="7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6" exitCode=0
Oct 30 22:47:18.352022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.351549    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"
Oct 30 22:47:18.352022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.351571    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" event={"ID":"f9d4d72e-f330-4e4c-b764-462f9e95871a","Type":"ContainerDied","Data":"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"}
Oct 30 22:47:18.352022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.351647    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj" event={"ID":"f9d4d72e-f330-4e4c-b764-462f9e95871a","Type":"ContainerDied","Data":"18812ca395cee64471eec74a52cbc1ebc7ec3c3c493b9faa3939b25eac2c1d5b"}
Oct 30 22:47:18.352022 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.351670    5510 scope.go:122] "RemoveContainer" containerID="7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"
Oct 30 22:47:18.360736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.360721    5510 scope.go:122] "RemoveContainer" containerID="7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"
Oct 30 22:47:18.360965 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:47:18.360945    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6\": container with ID starting with 7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6 not found: ID does not exist" containerID="7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"
Oct 30 22:47:18.361037 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.360971    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6"} err="failed to get container status \"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6\": rpc error: code = NotFound desc = could not find container \"7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6\": container with ID starting with 7deb5c087bc80929606184890d7c173c693aa3c20d82989409a7fb270ddfdba6 not found: ID does not exist"
Oct 30 22:47:18.509399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.509359    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:47:18.556697 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:18.556657    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b1b8bb-67cbfd4cd5-f2ftj"]
Oct 30 22:47:19.123377 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:19.123345    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d4d72e-f330-4e4c-b764-462f9e95871a" path="/var/lib/kubelet/pods/f9d4d72e-f330-4e4c-b764-462f9e95871a/volumes"
Oct 30 22:47:20.858570 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:20.858541    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:21.135218 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:21.135141    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:21.883143 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:21.883108    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:22.355930 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:22.355893    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:23.733629 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:23.733594    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:47:28.005796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:28.005745    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:28.167039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:28.167007    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:28.322906 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:28.322813    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:47:28.323100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:28.323055    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy" containerID="cri-o://4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a" gracePeriod=30
Oct 30 22:47:28.346859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:28.346817    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:33.574490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.574468    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:47:33.678322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678260    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678322 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678301    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678342    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678385    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678426    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678467    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678513 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678504    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678551    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678628    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\") pod \"44da6002-e234-42da-88c8-fb62db60ba1f\" (UID: \"44da6002-e234-42da-88c8-fb62db60ba1f\") "
Oct 30 22:47:33.678829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678668    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:33.678829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678686    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:33.679071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.678854    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:33.679071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.679035    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:33.679071 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.679041    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.679231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.679064    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:33.679231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.679089    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.679231 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.679108    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.680538 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.680512    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:33.680634 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.680520    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "kube-api-access-9w6m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:33.680751 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.680729    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:33.680835 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.680814    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo" pod "44da6002-e234-42da-88c8-fb62db60ba1f" (UID: "44da6002-e234-42da-88c8-fb62db60ba1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:33.779639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779608    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.779639 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779636    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44da6002-e234-42da-88c8-fb62db60ba1f-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.779857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779653    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6m8\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-kube-api-access-9w6m8\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.779857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779662    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44da6002-e234-42da-88c8-fb62db60ba1f-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.779857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779671    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44da6002-e234-42da-88c8-fb62db60ba1f-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.779857 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:33.779679    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44da6002-e234-42da-88c8-fb62db60ba1f-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:34.439543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.439505    5510 generic.go:356] "Generic (PLEG): container finished" podID="44da6002-e234-42da-88c8-fb62db60ba1f" containerID="4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a" exitCode=0
Oct 30 22:47:34.439543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.439537    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerDied","Data":"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"}
Oct 30 22:47:34.439849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.439565    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"
Oct 30 22:47:34.439849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.439588    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6" event={"ID":"44da6002-e234-42da-88c8-fb62db60ba1f","Type":"ContainerDied","Data":"624ac16d94762363fdddf25ea79e366c566a4df6830d8d5623764a7856bbf44d"}
Oct 30 22:47:34.439849 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.439620    5510 scope.go:122] "RemoveContainer" containerID="4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"
Oct 30 22:47:34.450082 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.449928    5510 scope.go:122] "RemoveContainer" containerID="4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"
Oct 30 22:47:34.457848 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.457834    5510 scope.go:122] "RemoveContainer" containerID="4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"
Oct 30 22:47:34.458093 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:47:34.458074    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a\": container with ID starting with 4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a not found: ID does not exist" containerID="4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"
Oct 30 22:47:34.458157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.458103    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a"} err="failed to get container status \"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a\": rpc error: code = NotFound desc = could not find container \"4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a\": container with ID starting with 4d812326d0072318faf7a2ff872b15b255b4492de89f5b892cb73b0bfb15188a not found: ID does not exist"
Oct 30 22:47:34.458157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.458121    5510 scope.go:122] "RemoveContainer" containerID="4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"
Oct 30 22:47:34.458356 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:47:34.458343    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e\": container with ID starting with 4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e not found: ID does not exist" containerID="4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"
Oct 30 22:47:34.458407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.458360    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e"} err="failed to get container status \"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e\": rpc error: code = NotFound desc = could not find container \"4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e\": container with ID starting with 4395fbf61ff3c5671b6c8c7b586d573fbfef46a08ad827c1b5513b840325e01e not found: ID does not exist"
Oct 30 22:47:34.638384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.638351    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:47:34.687818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:34.687790    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b1b8bb-openshift-default-65cc9c76857gqm6"]
Oct 30 22:47:35.123305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:35.123268    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" path="/var/lib/kubelet/pods/44da6002-e234-42da-88c8-fb62db60ba1f/volumes"
Oct 30 22:47:35.775660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:35.775626    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:38.081439 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:38.081403    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:38.714998 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:38.714964    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.435487 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:40.435455    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:47:43.345282 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:47:43.345243    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:47:46.639685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:46.639646    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:47.175061 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:47.175032    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:50.578459 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:50.578421    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:51.256355 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:51.256314    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:59.044040 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:59.044001    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"]
Oct 30 22:47:59.090189 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:59.090154    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bp92q"]
Oct 30 22:47:59.123706 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:47:59.123673    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a279d03-a238-445d-b073-a53a6f480279" path="/var/lib/kubelet/pods/6a279d03-a238-445d-b073-a53a6f480279/volumes"
Oct 30 22:48:01.742727 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:01.742695    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:12.015382 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:12.015355    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:48:12.449694 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:12.449614    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:48:13.279378 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:48:13.279343    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:48:13.423005 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:13.422968    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:18.448595 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.448563    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:48:18.448993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.448979    5510 cpu_manager.go:414] "containerMap: removing container" podUID="f9d4d72e-f330-4e4c-b764-462f9e95871a" containerName="maas-api"
Oct 30 22:48:18.449044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.448994    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d4d72e-f330-4e4c-b764-462f9e95871a" containerName="maas-api"
Oct 30 22:48:18.449044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449008    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449014    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449024    5510 cpu_manager.go:414] "containerMap: removing container" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449044 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449030    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449104    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449115    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="44da6002-e234-42da-88c8-fb62db60ba1f" containerName="istio-proxy"
Oct 30 22:48:18.449199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.449127    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d4d72e-f330-4e4c-b764-462f9e95871a" containerName="maas-api"
Oct 30 22:48:18.452094 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.452070    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.505502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.505474    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:48:18.541089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.541060    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:18.543247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543205    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.543383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543273    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.543383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543315    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.543467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543407    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.543467 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543459    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4dj\" (UniqueName: \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.543554 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.543489    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.546962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.546946    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5df7564a-kserve-self-signed-certs\""
Oct 30 22:48:18.645011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.644960    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645044    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4dj\" (UniqueName: \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645078    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645138    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645202 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645179    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645447 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645219    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645547    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645564    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.645751 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.645602    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.647370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.647352    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.647550 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.647532    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.698820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.698737    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4dj\" (UniqueName: \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\") pod \"e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") " pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:18.775591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:18.775561    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:19.169340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:19.169304    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:48:19.176600 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:48:19.176568    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3f6db42_ec07_4ca8_b59b_a9758e652488.slice/crio-c9cc453c198f368abb4f49e6bb33bd3ea1ef26d93d3bfd717e0eb2d3b36002e3 WatchSource:0}: Error finding container c9cc453c198f368abb4f49e6bb33bd3ea1ef26d93d3bfd717e0eb2d3b36002e3: Status 404 returned error can't find the container with id c9cc453c198f368abb4f49e6bb33bd3ea1ef26d93d3bfd717e0eb2d3b36002e3
Oct 30 22:48:19.687133 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:19.687092    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" event={"ID":"d3f6db42-ec07-4ca8-b59b-a9758e652488","Type":"ContainerStarted","Data":"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"}
Oct 30 22:48:19.687547 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:19.687143    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" event={"ID":"d3f6db42-ec07-4ca8-b59b-a9758e652488","Type":"ContainerStarted","Data":"c9cc453c198f368abb4f49e6bb33bd3ea1ef26d93d3bfd717e0eb2d3b36002e3"}
Oct 30 22:48:19.776431 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:19.776402    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:19.797896 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:19.797840    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" podStartSLOduration=1.7978252289999999 podStartE2EDuration="1.797825229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:19.796396469 +0000 UTC m=+3908.780303345" watchObservedRunningTime="2026-10-30 22:48:19.797825229 +0000 UTC m=+3908.781732083"
Oct 30 22:48:23.169169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:23.169132    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:48:28.776539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:28.776494    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:28.789074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:28.789048    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:29.765651 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:29.765623    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:48:39.107213 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:39.107183    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:43.279699 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:48:43.279664    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:48:52.111033 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.110997    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"]
Oct 30 22:48:52.114588 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.114555    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.167396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.167369    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"]
Oct 30 22:48:52.174399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.173872    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.174399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.173947    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfb4x\" (UniqueName: \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.174399 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.174006    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.211668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.211636    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:52.211878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.211859    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:52.211966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.211883    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:48:52.211966 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.211886    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:52.274519 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.274484    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.274660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.274528    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfb4x\" (UniqueName: \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.274660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.274591    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.275130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.275104    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.276865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.276847    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.324267 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.324239    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:52.327742 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.327716    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfb4x\" (UniqueName: \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\") pod \"maas-api-revoke-sub-452b6d7b-mv2gq\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.428297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.428219    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:52.477569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.477541    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:52.803851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.803817    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"]
Oct 30 22:48:52.868146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:52.868118    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq" event={"ID":"12acde89-b9cc-4b43-8b6e-69e008fd9850","Type":"ContainerStarted","Data":"ec7b6929f8922b0c4708869d8793a9755692eb459773f61517f02166e36da34a"}
Oct 30 22:48:53.235305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:53.235269    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:48:53.876813 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:53.876700    5510 generic.go:356] "Generic (PLEG): container finished" podID="12acde89-b9cc-4b43-8b6e-69e008fd9850" containerID="a0a54109e183ea79882c1bbe592312374e5f36fb68178938a2058ff986311c5d" exitCode=0
Oct 30 22:48:53.877002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:53.876813    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq" event={"ID":"12acde89-b9cc-4b43-8b6e-69e008fd9850","Type":"ContainerDied","Data":"a0a54109e183ea79882c1bbe592312374e5f36fb68178938a2058ff986311c5d"}
Oct 30 22:48:55.341958 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.341927    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:48:55.447567 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.447520    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\") pod \"12acde89-b9cc-4b43-8b6e-69e008fd9850\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") "
Oct 30 22:48:55.447749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.447637    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\") pod \"12acde89-b9cc-4b43-8b6e-69e008fd9850\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") "
Oct 30 22:48:55.447749 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.447668    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\" (UniqueName: \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\") pod \"12acde89-b9cc-4b43-8b6e-69e008fd9850\" (UID: \"12acde89-b9cc-4b43-8b6e-69e008fd9850\") "
Oct 30 22:48:55.447962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.447924    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca" pod "12acde89-b9cc-4b43-8b6e-69e008fd9850" (UID: "12acde89-b9cc-4b43-8b6e-69e008fd9850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:55.448085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.448068    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:55.449764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.449741    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca" pod "12acde89-b9cc-4b43-8b6e-69e008fd9850" (UID: "12acde89-b9cc-4b43-8b6e-69e008fd9850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:55.449879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.449835    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x" pod "12acde89-b9cc-4b43-8b6e-69e008fd9850" (UID: "12acde89-b9cc-4b43-8b6e-69e008fd9850"). InnerVolumeSpecName "kube-api-access-dfb4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:55.548872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.548760    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12acde89-b9cc-4b43-8b6e-69e008fd9850-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:55.548872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.548815    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfb4x\" (UniqueName: \"kubernetes.io/projected/12acde89-b9cc-4b43-8b6e-69e008fd9850-kube-api-access-dfb4x\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:55.922854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.921992    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq" event={"ID":"12acde89-b9cc-4b43-8b6e-69e008fd9850","Type":"ContainerDied","Data":"ec7b6929f8922b0c4708869d8793a9755692eb459773f61517f02166e36da34a"}
Oct 30 22:48:55.922854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.922084    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7b6929f8922b0c4708869d8793a9755692eb459773f61517f02166e36da34a"
Oct 30 22:48:55.922854 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:48:55.922001    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"
Oct 30 22:49:04.892183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:04.892133    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:08.753851 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:08.753814    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:49:11.371525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:11.371486    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:49:11.427344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:11.427310    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:13.279308 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:49:13.279269    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:49:15.515981 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.515949    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"]
Oct 30 22:49:15.516444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.516355    5510 cpu_manager.go:414] "containerMap: removing container" podUID="12acde89-b9cc-4b43-8b6e-69e008fd9850" containerName="revoke-keys"
Oct 30 22:49:15.516444 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.516366    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12acde89-b9cc-4b43-8b6e-69e008fd9850" containerName="revoke-keys"
Oct 30 22:49:15.516529 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.516457    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="12acde89-b9cc-4b43-8b6e-69e008fd9850" containerName="revoke-keys"
Oct 30 22:49:15.520057 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.520035    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.573177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.573142    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"]
Oct 30 22:49:15.576829 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.576800    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.576969 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.576925    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhpd\" (UniqueName: \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.577023 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.576996    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.617632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.617596    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:15.617632 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.617605    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:15.617870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.617615    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:15.617870 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.617646    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:49:15.677796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.677761    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.677971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.677840    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.677971 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.677908    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhpd\" (UniqueName: \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.678502 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.678478    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.680349 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.680320    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.732308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.732275    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhpd\" (UniqueName: \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\") pod \"maas-api-revoke-sub-3e31a506-tth2d\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:15.834627 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:15.834560    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:16.204287 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:16.204208    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"]
Oct 30 22:49:16.211887 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:49:16.211840    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598e8c8d_cb8d_4025_8e5b_5f462f9b2aaa.slice/crio-51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56 WatchSource:0}: Error finding container 51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56: Status 404 returned error can't find the container with id 51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56
Oct 30 22:49:17.048812 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:17.048763    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d" event={"ID":"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa","Type":"ContainerStarted","Data":"11c18035513610491bb6e991719515c2b6fcaa804d400ab0cb560ea970991cc1"}
Oct 30 22:49:17.049206 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:17.048856    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d" event={"ID":"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa","Type":"ContainerStarted","Data":"51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56"}
Oct 30 22:49:17.161164 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:17.161113    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d" podStartSLOduration=2.161099356 podStartE2EDuration="2.161099356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:17.159357751 +0000 UTC m=+3966.143264627" watchObservedRunningTime="2026-10-30 22:49:17.161099356 +0000 UTC m=+3966.145006243"
Oct 30 22:49:18.058803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:18.058581    5510 generic.go:356] "Generic (PLEG): container finished" podID="598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" containerID="11c18035513610491bb6e991719515c2b6fcaa804d400ab0cb560ea970991cc1" exitCode=0
Oct 30 22:49:18.058803 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:18.058627    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d" event={"ID":"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa","Type":"ContainerDied","Data":"11c18035513610491bb6e991719515c2b6fcaa804d400ab0cb560ea970991cc1"}
Oct 30 22:49:19.450169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.450139    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:19.538935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.538898    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\" (UniqueName: \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\") pod \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") "
Oct 30 22:49:19.539146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.538984    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\") pod \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") "
Oct 30 22:49:19.539146 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.539021    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\") pod \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\" (UID: \"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa\") "
Oct 30 22:49:19.539395 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.539364    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca" pod "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" (UID: "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:19.541319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.541287    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd" pod "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" (UID: "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa"). InnerVolumeSpecName "kube-api-access-qfhpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:19.541415 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.541328    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca" pod "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" (UID: "598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:19.640656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.640621    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhpd\" (UniqueName: \"kubernetes.io/projected/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-kube-api-access-qfhpd\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.640656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.640650    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.640656 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:19.640660    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:20.112830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:20.108451    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d" event={"ID":"598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa","Type":"ContainerDied","Data":"51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56"}
Oct 30 22:49:20.112830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:20.108522    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51aa6f27fe21327c5d7c179d5a37118047a8c68b7959ed60b142a7b159ea6c56"
Oct 30 22:49:20.112830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:20.109540    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"
Oct 30 22:49:23.953768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:23.953685    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:49:24.004379 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:24.004349    5510 scope.go:122] "RemoveContainer" containerID="2ff98ce977af01f990a5fe5d995a7ed93068adb8d1509ea855a8236c3530fdf4"
Oct 30 22:49:26.451830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.451770    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" podUID="d3f6db42-ec07-4ca8-b59b-a9758e652488" containerName="main" containerID="cri-o://aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782" gracePeriod=45
Oct 30 22:49:26.694115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.694090    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:49:26.810751 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810721    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.810967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810796    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.810967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810843    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.810967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810876    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.810967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810957    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.811209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.810983    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\" (UniqueName: \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\") pod \"d3f6db42-ec07-4ca8-b59b-a9758e652488\" (UID: \"d3f6db42-ec07-4ca8-b59b-a9758e652488\") "
Oct 30 22:49:26.811209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.811045    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:26.811340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.811222    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:26.811340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.811260    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:26.811456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.811337    5510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-model-cache\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.811456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.811367    5510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-home\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.813066 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.813038    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:26.813169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.813063    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "kube-api-access-rk4dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:26.813169 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.813078    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs" pod "d3f6db42-ec07-4ca8-b59b-a9758e652488" (UID: "d3f6db42-ec07-4ca8-b59b-a9758e652488"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:26.912158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.912125    5510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f6db42-ec07-4ca8-b59b-a9758e652488-tls-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.912158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.912152    5510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-dshm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.912158 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.912162    5510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f6db42-ec07-4ca8-b59b-a9758e652488-tmp-dir\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.912397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:26.912172    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4dj\" (UniqueName: \"kubernetes.io/projected/d3f6db42-ec07-4ca8-b59b-a9758e652488-kube-api-access-rk4dj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:27.141187 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.141155    5510 generic.go:356] "Generic (PLEG): container finished" podID="d3f6db42-ec07-4ca8-b59b-a9758e652488" containerID="aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782" exitCode=0
Oct 30 22:49:27.141340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.141206    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"
Oct 30 22:49:27.141340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.141245    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" event={"ID":"d3f6db42-ec07-4ca8-b59b-a9758e652488","Type":"ContainerDied","Data":"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"}
Oct 30 22:49:27.141340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.141296    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k" event={"ID":"d3f6db42-ec07-4ca8-b59b-a9758e652488","Type":"ContainerDied","Data":"c9cc453c198f368abb4f49e6bb33bd3ea1ef26d93d3bfd717e0eb2d3b36002e3"}
Oct 30 22:49:27.141340 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.141317    5510 scope.go:122] "RemoveContainer" containerID="aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"
Oct 30 22:49:27.151088 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.151063    5510 scope.go:122] "RemoveContainer" containerID="aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"
Oct 30 22:49:27.151372 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:49:27.151353    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782\": container with ID starting with aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782 not found: ID does not exist" containerID="aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"
Oct 30 22:49:27.151451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.151385    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782"} err="failed to get container status \"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782\": rpc error: code = NotFound desc = could not find container \"aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782\": container with ID starting with aa4e3888c4d8c3f9f9281cc4c1f2e51396593570b23aebeb9807ef9cae35d782 not found: ID does not exist"
Oct 30 22:49:27.294457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.294421    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:49:27.340675 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:27.340640    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5df7564a-kserve-7dddcd86f7-kgc6k"]
Oct 30 22:49:29.128359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:29.128315    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f6db42-ec07-4ca8-b59b-a9758e652488" path="/var/lib/kubelet/pods/d3f6db42-ec07-4ca8-b59b-a9758e652488/volumes"
Oct 30 22:49:33.644776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:33.644742    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:49:39.401147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:39.401103    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:41.947591 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.947551    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"]
Oct 30 22:49:41.948263 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948237    5510 cpu_manager.go:414] "containerMap: removing container" podUID="d3f6db42-ec07-4ca8-b59b-a9758e652488" containerName="main"
Oct 30 22:49:41.948316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948269    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f6db42-ec07-4ca8-b59b-a9758e652488" containerName="main"
Oct 30 22:49:41.948316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948301    5510 cpu_manager.go:414] "containerMap: removing container" podUID="598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" containerName="revoke-keys"
Oct 30 22:49:41.948316 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948310    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" containerName="revoke-keys"
Oct 30 22:49:41.948455 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948440    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f6db42-ec07-4ca8-b59b-a9758e652488" containerName="main"
Oct 30 22:49:41.948499 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.948463    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" containerName="revoke-keys"
Oct 30 22:49:41.952351 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:41.952328    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.047317 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.047288    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:42.047493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.047374    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:42.047493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.047396    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:42.047493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.047457    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gdmfg\""
Oct 30 22:49:42.049030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.049010    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"]
Oct 30 22:49:42.095865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.095834    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.096056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.095874    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfxm\" (UniqueName: \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.096056 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.095892    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.197312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.197276    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.197312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.197315    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfxm\" (UniqueName: \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.197553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.197336    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.197951 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.197904    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.199698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.199677    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.251963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.251933    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfxm\" (UniqueName: \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\") pod \"maas-api-revoke-sub-14c12f66-q9llp\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.265524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.265485    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:42.632113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:42.632075    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"]
Oct 30 22:49:42.638810 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:49:42.638758    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b50c1a_afa8_4162_8b0d_1582b0c21c01.slice/crio-cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751 WatchSource:0}: Error finding container cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751: Status 404 returned error can't find the container with id cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751
Oct 30 22:49:43.267356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:43.267282    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp" event={"ID":"89b50c1a-afa8-4162-8b0d-1582b0c21c01","Type":"ContainerStarted","Data":"e8adc0de6d929de7e9326db1815a6a72b2478eb361c87e26fccd248d1718b6df"}
Oct 30 22:49:43.267356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:43.267334    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp" event={"ID":"89b50c1a-afa8-4162-8b0d-1582b0c21c01","Type":"ContainerStarted","Data":"cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751"}
Oct 30 22:49:43.281253 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:49:43.281222    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:49:43.380011 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:43.379964    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp" podStartSLOduration=2.37995009 podStartE2EDuration="2.37995009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:43.379164547 +0000 UTC m=+3992.363071443" watchObservedRunningTime="2026-10-30 22:49:43.37995009 +0000 UTC m=+3992.363856985"
Oct 30 22:49:44.273865 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:44.273830    5510 generic.go:356] "Generic (PLEG): container finished" podID="89b50c1a-afa8-4162-8b0d-1582b0c21c01" containerID="e8adc0de6d929de7e9326db1815a6a72b2478eb361c87e26fccd248d1718b6df" exitCode=0
Oct 30 22:49:44.274327 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:44.273914    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp" event={"ID":"89b50c1a-afa8-4162-8b0d-1582b0c21c01","Type":"ContainerDied","Data":"e8adc0de6d929de7e9326db1815a6a72b2478eb361c87e26fccd248d1718b6df"}
Oct 30 22:49:45.403702 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.403677    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:45.528386 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.528248    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\" (UniqueName: \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\") pod \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") "
Oct 30 22:49:45.528539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.528452    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\") pod \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") "
Oct 30 22:49:45.528975 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.528931    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\") pod \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\" (UID: \"89b50c1a-afa8-4162-8b0d-1582b0c21c01\") "
Oct 30 22:49:45.529442 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.529410    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca" pod "89b50c1a-afa8-4162-8b0d-1582b0c21c01" (UID: "89b50c1a-afa8-4162-8b0d-1582b0c21c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:45.531290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.531258    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca" pod "89b50c1a-afa8-4162-8b0d-1582b0c21c01" (UID: "89b50c1a-afa8-4162-8b0d-1582b0c21c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:45.531290 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.531278    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm" pod "89b50c1a-afa8-4162-8b0d-1582b0c21c01" (UID: "89b50c1a-afa8-4162-8b0d-1582b0c21c01"). InnerVolumeSpecName "kube-api-access-cqfxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.629926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.629890    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-openshift-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.629926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.629920    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfxm\" (UniqueName: \"kubernetes.io/projected/89b50c1a-afa8-4162-8b0d-1582b0c21c01-kube-api-access-cqfxm\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:45.629926 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:45.629931    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b50c1a-afa8-4162-8b0d-1582b0c21c01-maas-api-xks-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:46.283900 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:46.283865    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp" event={"ID":"89b50c1a-afa8-4162-8b0d-1582b0c21c01","Type":"ContainerDied","Data":"cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751"}
Oct 30 22:49:46.284078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:46.283900    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"
Oct 30 22:49:46.284078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:46.283913    5510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad5183dad0c6585e6e2cd6a3d3a3258cb691866c2ba80b6cb856bab1306d751"
Oct 30 22:49:54.771115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:54.771081    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:49:55.266743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.266708    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:49:55.269599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.269562    5510 cpu_manager.go:414] "containerMap: removing container" podUID="89b50c1a-afa8-4162-8b0d-1582b0c21c01" containerName="revoke-keys"
Oct 30 22:49:55.269599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.269594    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b50c1a-afa8-4162-8b0d-1582b0c21c01" containerName="revoke-keys"
Oct 30 22:49:55.269915 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.269745    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b50c1a-afa8-4162-8b0d-1582b0c21c01" containerName="revoke-keys"
Oct 30 22:49:55.273845 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.273814    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.322722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.322686    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:55.368961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.368929    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:55.368961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.368929    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:49:55.368961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.368929    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:49:55.369197 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.369032    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rsvrw\""
Oct 30 22:49:55.370775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.370752    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:49:55.419823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.419768    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.419987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.419903    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.419987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.419984    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2c5g\" (UniqueName: \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.420093 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.420074    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.521566 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.521480    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.521744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.521580    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2c5g\" (UniqueName: \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.521744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.521614    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.521744 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.521654    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.522305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.522285    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.523918 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.523896    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.524000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.523945    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.574617 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.574591    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2c5g\" (UniqueName: \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\") pod \"maas-api-797f8b87df-66r2c\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") " pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.591492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.591467    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:55.960002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:55.959968    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:49:55.968433 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:49:55.968406    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb5242a_c7eb_422c_82e7_bc4b2d2d01f7.slice/crio-53e6500b35029ab5a8e02a29cd24ea5a456a76ec27f6c62f7ed4f96ce1e1e238 WatchSource:0}: Error finding container 53e6500b35029ab5a8e02a29cd24ea5a456a76ec27f6c62f7ed4f96ce1e1e238: Status 404 returned error can't find the container with id 53e6500b35029ab5a8e02a29cd24ea5a456a76ec27f6c62f7ed4f96ce1e1e238
Oct 30 22:49:56.330954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:56.330867    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" event={"ID":"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7","Type":"ContainerStarted","Data":"53e6500b35029ab5a8e02a29cd24ea5a456a76ec27f6c62f7ed4f96ce1e1e238"}
Oct 30 22:49:57.054524 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.054483    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"]
Oct 30 22:49:57.075534 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.075505    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r5w84"]
Oct 30 22:49:57.123799 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.123713    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6916db88-cc98-40d5-aefb-c901b0a1386e" path="/var/lib/kubelet/pods/6916db88-cc98-40d5-aefb-c901b0a1386e/volumes"
Oct 30 22:49:57.336820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.336766    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" event={"ID":"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7","Type":"ContainerStarted","Data":"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"}
Oct 30 22:49:57.336999 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.336844    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:49:57.447053 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:49:57.446964    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" podStartSLOduration=1.5427805239999999 podStartE2EDuration="2.446949293s" totalImagesPullingTime="904.168769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:57.445035276 +0000 UTC m=+4006.428942152" watchObservedRunningTime="2026-10-30 22:49:57.446949293 +0000 UTC m=+4006.430856211"
Oct 30 22:50:02.103715 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.103669    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:02.625315 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.625281    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:50:02.629335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.629310    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.680989 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.680954    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8756f17a-openshift-default-dockercfg-4qzhq\""
Oct 30 22:50:02.730344 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.730310    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:50:02.733622 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.733576    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:02.797512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797468    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797545    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797611    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797648    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797673    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797703    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.797763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797761    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226lj\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.798238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797857    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.798238 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.797916    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.898693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898597    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.898693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898650    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.898693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898681    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898709    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898745    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226lj\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898827    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898877    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898913    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899014 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.898954    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.899118    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899319 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.899145    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.899359    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.899402    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.899693 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.899673    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.901226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.901204    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.901426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.901407    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.953129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.953082    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.953361 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.953195    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226lj\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\") pod \"e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") " pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:02.954454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:02.954425    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:03.082605 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.082557    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:50:03.082831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.082796    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" podUID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" containerName="maas-api" containerID="cri-o://21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83" gracePeriod=30
Oct 30 22:50:03.089537 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.089509    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:50:03.126687 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.126651    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"]
Oct 30 22:50:03.130248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.130211    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.186147 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.186113    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"]
Oct 30 22:50:03.243114 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.243082    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:03.304099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.303699    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbk5\" (UniqueName: \"kubernetes.io/projected/049a576d-50e6-464b-8e8e-f58da7c630eb-kube-api-access-zrbk5\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.304099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.303764    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-maas-api-tls\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.304099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.303943    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a576d-50e6-464b-8e8e-f58da7c630eb-openshift-service-ca\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.304313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.304155    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-metrics-certs\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.348664 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.348637    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:50:03.365248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.365216    5510 generic.go:356] "Generic (PLEG): container finished" podID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" containerID="21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83" exitCode=0
Oct 30 22:50:03.365401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.365267    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"
Oct 30 22:50:03.365401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.365310    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" event={"ID":"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7","Type":"ContainerDied","Data":"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"}
Oct 30 22:50:03.365401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.365354    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-797f8b87df-66r2c" event={"ID":"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7","Type":"ContainerDied","Data":"53e6500b35029ab5a8e02a29cd24ea5a456a76ec27f6c62f7ed4f96ce1e1e238"}
Oct 30 22:50:03.365401 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.365378    5510 scope.go:122] "RemoveContainer" containerID="21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"
Oct 30 22:50:03.374237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.374213    5510 scope.go:122] "RemoveContainer" containerID="21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"
Oct 30 22:50:03.374504 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:50:03.374485    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83\": container with ID starting with 21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83 not found: ID does not exist" containerID="21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"
Oct 30 22:50:03.374569 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.374515    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83"} err="failed to get container status \"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83\": rpc error: code = NotFound desc = could not find container \"21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83\": container with ID starting with 21b889192c8103636b9508c0c030ef86e6b733680d5db30da53b3c53a37d1f83 not found: ID does not exist"
Oct 30 22:50:03.405525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405493    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\") pod \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") "
Oct 30 22:50:03.405674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405539    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\") pod \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") "
Oct 30 22:50:03.405674 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405614    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\" (UniqueName: \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\") pod \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") "
Oct 30 22:50:03.405769 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405686    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\") pod \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\" (UID: \"cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7\") "
Oct 30 22:50:03.405931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405908    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-metrics-certs\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.406019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.405989    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca" pod "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" (UID: "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:03.406077 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.406043    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbk5\" (UniqueName: \"kubernetes.io/projected/049a576d-50e6-464b-8e8e-f58da7c630eb-kube-api-access-zrbk5\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.406132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.406091    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-maas-api-tls\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.406132 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.406109    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a576d-50e6-464b-8e8e-f58da7c630eb-openshift-service-ca\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.406248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.406169    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.406927 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.406894    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a576d-50e6-464b-8e8e-f58da7c630eb-openshift-service-ca\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.407878 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.407857    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls" pod "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" (UID: "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:03.407962 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.407939    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g" pod "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" (UID: "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7"). InnerVolumeSpecName "kube-api-access-n2c5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:03.408220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.408190    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs" pod "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" (UID: "cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:03.408597 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.408568    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-maas-api-tls\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.408680 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.408582    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/049a576d-50e6-464b-8e8e-f58da7c630eb-metrics-certs\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.461539 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.461508    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbk5\" (UniqueName: \"kubernetes.io/projected/049a576d-50e6-464b-8e8e-f58da7c630eb-kube-api-access-zrbk5\") pod \"maas-api-5fd44bb67c-h2qj5\" (UID: \"049a576d-50e6-464b-8e8e-f58da7c630eb\") " pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.478410 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.478373    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:03.507756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.507717    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2c5g\" (UniqueName: \"kubernetes.io/projected/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-kube-api-access-n2c5g\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.507756 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.507753    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.508006 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.507765    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.639943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.639908    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:50:03.647122 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:50:03.647091    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d0c77e_4984_41c3_8cdb_4db78e91da88.slice/crio-0444b8e761ae95ee66f000c90041b937bf67d268989610dde1f3441d70dba95f WatchSource:0}: Error finding container 0444b8e761ae95ee66f000c90041b937bf67d268989610dde1f3441d70dba95f: Status 404 returned error can't find the container with id 0444b8e761ae95ee66f000c90041b937bf67d268989610dde1f3441d70dba95f
Oct 30 22:50:03.649365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.649332    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:03.649452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.649399    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:03.649452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.649431    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:03.785733 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.785686    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:50:03.791321 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.791283    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-797f8b87df-66r2c"]
Oct 30 22:50:03.853457 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:03.853412    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"]
Oct 30 22:50:03.861332 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:50:03.861299    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049a576d_50e6_464b_8e8e_f58da7c630eb.slice/crio-2bf7bf2c4d2bad68ec2fa1c4491447684a105770df09d7e0e27c68520cba185f WatchSource:0}: Error finding container 2bf7bf2c4d2bad68ec2fa1c4491447684a105770df09d7e0e27c68520cba185f: Status 404 returned error can't find the container with id 2bf7bf2c4d2bad68ec2fa1c4491447684a105770df09d7e0e27c68520cba185f
Oct 30 22:50:04.374611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:04.374562    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerStarted","Data":"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"}
Oct 30 22:50:04.375228 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:04.374681    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerStarted","Data":"0444b8e761ae95ee66f000c90041b937bf67d268989610dde1f3441d70dba95f"}
Oct 30 22:50:04.380575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:04.380539    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5" event={"ID":"049a576d-50e6-464b-8e8e-f58da7c630eb","Type":"ContainerStarted","Data":"2bf7bf2c4d2bad68ec2fa1c4491447684a105770df09d7e0e27c68520cba185f"}
Oct 30 22:50:04.486872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:04.486818    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podStartSLOduration=2.486804103 podStartE2EDuration="2.486804103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:04.485200541 +0000 UTC m=+4013.469107431" watchObservedRunningTime="2026-10-30 22:50:04.486804103 +0000 UTC m=+4013.470711016"
Oct 30 22:50:05.124134 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.124047    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" path="/var/lib/kubelet/pods/cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7/volumes"
Oct 30 22:50:05.243209 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.243171    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:05.248229 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.248198    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:05.389092 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.388986    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5" event={"ID":"049a576d-50e6-464b-8e8e-f58da7c630eb","Type":"ContainerStarted","Data":"d0425a82bdff8f5be7a225b1badca900b9568f5db7a2660af7d989b33eaf6bb9"}
Oct 30 22:50:05.389577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.389155    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:05.389577 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.389215    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:05.390866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.390843    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:05.626764 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:05.626695    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5" podStartSLOduration=1.663633732 podStartE2EDuration="2.626674576s" totalImagesPullingTime="963.040844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:05.624744912 +0000 UTC m=+4014.608651793" watchObservedRunningTime="2026-10-30 22:50:05.626674576 +0000 UTC m=+4014.610581452"
Oct 30 22:50:06.607237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:06.607159    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.781802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:06.781742    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:06.940561 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:06.940417    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:09.756679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:09.756650    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:09.910641 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:09.910588    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.067963 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.067863    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:10.152937 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.152906    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:10.153314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.153302    5510 cpu_manager.go:414] "containerMap: removing container" podUID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" containerName="maas-api"
Oct 30 22:50:10.153359 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.153315    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" containerName="maas-api"
Oct 30 22:50:10.153421 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.153412    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb5242a-c7eb-422c-82e7-bc4b2d2d01f7" containerName="maas-api"
Oct 30 22:50:10.156768 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.156748    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.174954 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.174929    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqlw\" (UniqueName: \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.175091 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.175070    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.258904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.258864    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8756f17a-dockercfg-fjwx4\""
Oct 30 22:50:10.258904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.258872    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8756f17a\""
Oct 30 22:50:10.260468 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.260446    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:10.276381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.276352    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.276503 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.276458    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqlw\" (UniqueName: \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.277102 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.277085    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.330220 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.330157    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqlw\" (UniqueName: \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\") pod \"payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.469337 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.469297    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:10.837090 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:10.837053    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:10.844742 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:50:10.844713    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b5c603_f4ed_4b6a_b0a5_e54c0ca8dfe7.slice/crio-f00250413d832b8de679ccd211bd7ee4fb3e83818b98b6350449c3ea06b7c8ab WatchSource:0}: Error finding container f00250413d832b8de679ccd211bd7ee4fb3e83818b98b6350449c3ea06b7c8ab: Status 404 returned error can't find the container with id f00250413d832b8de679ccd211bd7ee4fb3e83818b98b6350449c3ea06b7c8ab
Oct 30 22:50:11.400704 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:11.400675    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fd44bb67c-h2qj5"
Oct 30 22:50:11.419654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:11.419625    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" event={"ID":"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7","Type":"ContainerStarted","Data":"480d80210c2f499c76c2a8d81b8fe88d2263a90550216ffa65ce36c4bb9b2a09"}
Oct 30 22:50:11.419837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:11.419671    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" event={"ID":"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7","Type":"ContainerStarted","Data":"f00250413d832b8de679ccd211bd7ee4fb3e83818b98b6350449c3ea06b7c8ab"}
Oct 30 22:50:11.419837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:11.419714    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:11.617882 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:11.617821    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" podStartSLOduration=1.6178007810000001 podStartE2EDuration="1.617800781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:11.616456019 +0000 UTC m=+4020.600362901" watchObservedRunningTime="2026-10-30 22:50:11.617800781 +0000 UTC m=+4020.601707662"
Oct 30 22:50:13.284219 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:50:13.284184    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:50:14.817688 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:14.817657    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:17.551171 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:17.551135    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:50:19.376808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:19.374878    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:22.428717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:22.428684    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:24.028135 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:24.028071    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:26.050411 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.050379    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"]
Oct 30 22:50:26.096722 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.096694    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hgqrn"]
Oct 30 22:50:26.511860 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.511823    5510 generic.go:356] "Generic (PLEG): container finished" podID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerID="9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8" exitCode=0
Oct 30 22:50:26.512046 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.511900    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerDied","Data":"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"}
Oct 30 22:50:26.512282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.512267    5510 scope.go:122] "RemoveContainer" containerID="9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"
Oct 30 22:50:26.513396 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.513370    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:26.513476 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.513433    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:26.513517 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:26.513480    5510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 22:50:27.123576 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:27.123542    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ce9ab9-449a-4823-98b2-d2ea261db9b8" path="/var/lib/kubelet/pods/a5ce9ab9-449a-4823-98b2-d2ea261db9b8/volumes"
Oct 30 22:50:27.519030 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:27.518995    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerStarted","Data":"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"}
Oct 30 22:50:28.243360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:28.243327    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:28.243766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:28.243615    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:50:28.243766 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:28.243662    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:50:28.595452 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:28.595370    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:29.071802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:29.058562    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:29.244309 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:29.244280    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:50:29.244712 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:29.244329    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:50:30.244100 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:30.244054    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:50:30.244289 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:30.244113    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:50:31.243774 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:31.243737    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:50:31.244223 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:31.243815    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:50:32.243943 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:32.243912    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:50:32.244335 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:32.243961    5510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:50:33.247314 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:33.247283    5510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:33.247717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:33.247693    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:33.248409 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:33.248388    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:50:33.859423 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:33.859386    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:35.001729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:35.001693    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:35.154933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:35.154898    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:38.354516 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:38.354483    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:39.739004 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:39.738969    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:42.871036 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:42.871002    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:43.280010 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:50:43.279961    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:50:46.221530 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:46.221492    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:46.373657 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:46.373620    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:47.368089 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:47.368059    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:49.454543 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:49.454507    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:49.723575 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:49.723542    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:49.871065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:49.871033    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:51.754383 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:51.754305    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:54.455802 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.455744    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" podUID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" containerName="payload-pre-processing" containerID="cri-o://480d80210c2f499c76c2a8d81b8fe88d2263a90550216ffa65ce36c4bb9b2a09" gracePeriod=25
Oct 30 22:50:54.651177 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.651146    5510 generic.go:356] "Generic (PLEG): container finished" podID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" containerID="480d80210c2f499c76c2a8d81b8fe88d2263a90550216ffa65ce36c4bb9b2a09" exitCode=0
Oct 30 22:50:54.651370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.651188    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" event={"ID":"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7","Type":"ContainerDied","Data":"480d80210c2f499c76c2a8d81b8fe88d2263a90550216ffa65ce36c4bb9b2a09"}
Oct 30 22:50:54.705493 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.705463    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:54.759354 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.759321    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\") pod \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") "
Oct 30 22:50:54.759531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.759407    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\" (UniqueName: \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\") pod \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\" (UID: \"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7\") "
Oct 30 22:50:54.759755 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.759729    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume" pod "84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" (UID: "84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:54.761456 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.761434    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw" pod "84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" (UID: "84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7"). InnerVolumeSpecName "kube-api-access-qjqlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:54.860820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.860764    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.860820 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:54.860817    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqlw\" (UniqueName: \"kubernetes.io/projected/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7-kube-api-access-qjqlw\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:55.658796 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.658758    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"
Oct 30 22:50:55.659265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.658814    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6" event={"ID":"84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7","Type":"ContainerDied","Data":"f00250413d832b8de679ccd211bd7ee4fb3e83818b98b6350449c3ea06b7c8ab"}
Oct 30 22:50:55.659265 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.658886    5510 scope.go:122] "RemoveContainer" containerID="480d80210c2f499c76c2a8d81b8fe88d2263a90550216ffa65ce36c4bb9b2a09"
Oct 30 22:50:55.701021 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.700991    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:55.812031 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.811993    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:55.815184 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:55.815156    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8756f17a-d6578dd8c-2gwm6"]
Oct 30 22:50:57.124248 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:57.124211    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" path="/var/lib/kubelet/pods/84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7/volumes"
Oct 30 22:50:57.145441 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:57.145411    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:57.416700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:57.416572    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:59.223929 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:59.223899    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:50:59.224384 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:59.224107    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" containerID="cri-o://9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e" gracePeriod=30
Oct 30 22:50:59.266001 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:50:59.265967    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:00.522668 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:00.522631    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.153961 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:01.153926    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.355903 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:01.355859    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:02.042490 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:02.042449    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:51:03.250113 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:03.250075    5510 patch_prober.go:29] interesting pod/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:51:03.250510 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:03.250142    5510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:51:04.475075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.475043    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:51:04.576065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.575969    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576026    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576077    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576112    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576241    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576268    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576328 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576310    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:04.576620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576437    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:04.576620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576526    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:04.576620 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576535    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:04.576801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576633    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576693    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576801 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576731    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\") pod \"86d0c77e-4984-41c3-8cdb-4db78e91da88\" (UID: \"86d0c77e-4984-41c3-8cdb-4db78e91da88\") "
Oct 30 22:51:04.576960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.576900    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:04.577188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.577163    5510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d0c77e-4984-41c3-8cdb-4db78e91da88-istiod-ca-cert\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.577302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.577192    5510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.577302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.577206    5510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-workload-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.577302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.577219    5510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-credential-socket\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.577302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.577233    5510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-data\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.578282 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.578258    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:04.578904 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.578875    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:04.579039 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.579015    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:04.579085 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.579035    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj" pod "86d0c77e-4984-41c3-8cdb-4db78e91da88" (UID: "86d0c77e-4984-41c3-8cdb-4db78e91da88"). InnerVolumeSpecName "kube-api-access-226lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:04.678341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.678296    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226lj\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-kube-api-access-226lj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.678341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.678336    5510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-envoy\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.678341 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.678350    5510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-token\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.678603 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.678362    5510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d0c77e-4984-41c3-8cdb-4db78e91da88-istio-podinfo\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:04.705814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.705766    5510 generic.go:356] "Generic (PLEG): container finished" podID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerID="9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e" exitCode=0
Oct 30 22:51:04.705995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.705818    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerDied","Data":"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"}
Oct 30 22:51:04.705995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.705850    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"
Oct 30 22:51:04.705995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.705878    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78" event={"ID":"86d0c77e-4984-41c3-8cdb-4db78e91da88","Type":"ContainerDied","Data":"0444b8e761ae95ee66f000c90041b937bf67d268989610dde1f3441d70dba95f"}
Oct 30 22:51:04.705995 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.705903    5510 scope.go:122] "RemoveContainer" containerID="9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"
Oct 30 22:51:04.715599 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.715582    5510 scope.go:122] "RemoveContainer" containerID="9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"
Oct 30 22:51:04.723694 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.723673    5510 scope.go:122] "RemoveContainer" containerID="9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"
Oct 30 22:51:04.724010 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:04.723991    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e\": container with ID starting with 9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e not found: ID does not exist" containerID="9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"
Oct 30 22:51:04.724078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.724022    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e"} err="failed to get container status \"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e\": rpc error: code = NotFound desc = could not find container \"9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e\": container with ID starting with 9841b6ee6df7e90e10792dee3d121eca75a2f41c57a9a605c3f2250b59e7f09e not found: ID does not exist"
Oct 30 22:51:04.724078 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.724041    5510 scope.go:122] "RemoveContainer" containerID="9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"
Oct 30 22:51:04.724269 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:04.724248    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8\": container with ID starting with 9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8 not found: ID does not exist" containerID="9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"
Oct 30 22:51:04.724308 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.724276    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8"} err="failed to get container status \"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8\": rpc error: code = NotFound desc = could not find container \"9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8\": container with ID starting with 9ee55f1c3cae1587ff933b154bcd52d287747fecdb5504789443cf9ce74e81b8 not found: ID does not exist"
Oct 30 22:51:04.859985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.859903    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:51:04.905880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:04.905841    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8756f17a-openshift-default-74b9686b67-vsj78"]
Oct 30 22:51:05.123590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:05.123514    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" path="/var/lib/kubelet/pods/86d0c77e-4984-41c3-8cdb-4db78e91da88/volumes"
Oct 30 22:51:05.213153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:05.213116    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:05.413426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:05.413334    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:05.575763 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:05.575732    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.416660 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.416622    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.573593 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.573551    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.741816 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.741760    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:51:08.742312 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742294    5510 cpu_manager.go:414] "containerMap: removing container" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.742381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742314    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.742381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742328    5510 cpu_manager.go:414] "containerMap: removing container" podUID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" containerName="payload-pre-processing"
Oct 30 22:51:08.742381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742333    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" containerName="payload-pre-processing"
Oct 30 22:51:08.742500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742421    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.742500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742433    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.742500 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.742439    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b5c603-f4ed-4b6a-b0a5-e54c0ca8dfe7" containerName="payload-pre-processing"
Oct 30 22:51:08.747225 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.747204    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.798818 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.798772    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:51:08.819247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.819206    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.819247 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.819248    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nnn\" (UniqueName: \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.842807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.842742    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8a1d54\""
Oct 30 22:51:08.842807 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.842740    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-8a1d54-dockercfg-fsctb\""
Oct 30 22:51:08.909614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.909578    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:51:08.910075 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.910060    5510 cpu_manager.go:414] "containerMap: removing container" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.910144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.910076    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d0c77e-4984-41c3-8cdb-4db78e91da88" containerName="istio-proxy"
Oct 30 22:51:08.913366 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.913345    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:08.920222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.920127    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.920222 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.920183    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nnn\" (UniqueName: \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.920972 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.920947    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:08.967811 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.967743    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:51:08.973407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:08.973379    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nnn\" (UniqueName: \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\") pod \"payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:09.021381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.021299    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.021381 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.021341    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4wj\" (UniqueName: \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.062052 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.062014    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:09.122736 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.122706    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.122887 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.122753    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4wj\" (UniqueName: \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.124074 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.124044    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.176770 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.176738    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4wj\" (UniqueName: \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\") pod \"payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.227515 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.227476    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.433512 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.433474    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:51:09.440679 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:51:09.440651    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb934e5de_3037_4684_a46d_39bc3d01e569.slice/crio-8c1b7f6ec5cdeba4424d0822f70b2d9646b2e3cb55777cad3a688e190e91834d WatchSource:0}: Error finding container 8c1b7f6ec5cdeba4424d0822f70b2d9646b2e3cb55777cad3a688e190e91834d: Status 404 returned error can't find the container with id 8c1b7f6ec5cdeba4424d0822f70b2d9646b2e3cb55777cad3a688e190e91834d
Oct 30 22:51:09.597650 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.597566    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:51:09.605317 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:51:09.605287    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48449120_fac3_47bf_a265_25b85cca8fcd.slice/crio-8124d3eadd75cbc1ece46d2d234a996f63ec64b907a59b2e4614b7aff771313c WatchSource:0}: Error finding container 8124d3eadd75cbc1ece46d2d234a996f63ec64b907a59b2e4614b7aff771313c: Status 404 returned error can't find the container with id 8124d3eadd75cbc1ece46d2d234a996f63ec64b907a59b2e4614b7aff771313c
Oct 30 22:51:09.732481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.732348    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" event={"ID":"b934e5de-3037-4684-a46d-39bc3d01e569","Type":"ContainerStarted","Data":"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"}
Oct 30 22:51:09.732481 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.732411    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" event={"ID":"b934e5de-3037-4684-a46d-39bc3d01e569","Type":"ContainerStarted","Data":"8c1b7f6ec5cdeba4424d0822f70b2d9646b2e3cb55777cad3a688e190e91834d"}
Oct 30 22:51:09.732729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.732521    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:09.734065 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.734039    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" event={"ID":"48449120-fac3-47bf-a265-25b85cca8fcd","Type":"ContainerStarted","Data":"e5e7224fca6e8789d70bcba35fb65bd7eb0da1902199cb3f6ab5cd65f2f846b8"}
Oct 30 22:51:09.734178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.734078    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" event={"ID":"48449120-fac3-47bf-a265-25b85cca8fcd","Type":"ContainerStarted","Data":"8124d3eadd75cbc1ece46d2d234a996f63ec64b907a59b2e4614b7aff771313c"}
Oct 30 22:51:09.734178 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.734109    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:09.840226 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.840169    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" podStartSLOduration=1.84015599 podStartE2EDuration="1.84015599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:09.838864623 +0000 UTC m=+4078.822771500" watchObservedRunningTime="2026-10-30 22:51:09.84015599 +0000 UTC m=+4078.824062865"
Oct 30 22:51:09.948814 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:09.948247    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" podStartSLOduration=1.948220811 podStartE2EDuration="1.948220811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:09.946076966 +0000 UTC m=+4078.929983843" watchObservedRunningTime="2026-10-30 22:51:09.948220811 +0000 UTC m=+4078.932127688"
Oct 30 22:51:11.014944 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.014909    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:51:11.018514 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.018487    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.070332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.070308    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8a1d54-serving-cert\""
Oct 30 22:51:11.123426 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.123397    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:51:11.141472 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.141439    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvvv\" (UniqueName: \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.141615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.141484    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.141615 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.141581    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.141719 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.141622    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.216988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.216950    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.242837 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.242804    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvvv\" (UniqueName: \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.242960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.242844    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.242960 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.242872    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.243203 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.243028    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.243443 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.243422    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.245302 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.245284    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.290454 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.290380    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8a1d54-serving-cert\""
Oct 30 22:51:11.293489 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:11.293458    5510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-serving-cert: secret "maas-api-e2e-authcfg-http-8a1d54-serving-cert" not found
Oct 30 22:51:11.293631 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:11.293539    5510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls podName:576a17d1-cf8c-46d2-9001-5bde4c055cec nodeName:}" failed. No retries permitted until 2026-10-30 22:51:11.793518843 +0000 UTC m=+4080.777425698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls") pod "maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" (UID: "576a17d1-cf8c-46d2-9001-5bde4c055cec") : secret "maas-api-e2e-authcfg-http-8a1d54-serving-cert" not found
Oct 30 22:51:11.300297 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.300273    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvvv\" (UniqueName: \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.373076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.373046    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.523553 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.523510    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.851821 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.851765    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.854118 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.854088    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:11.931654 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:11.931618    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:12.304261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:12.304226    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:51:12.311681 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:51:12.311654    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576a17d1_cf8c_46d2_9001_5bde4c055cec.slice/crio-bab598c0dc3b79f6cd4b318fbf4f459fade17b289a29b0a2e9a9a8c6040d7a8b WatchSource:0}: Error finding container bab598c0dc3b79f6cd4b318fbf4f459fade17b289a29b0a2e9a9a8c6040d7a8b: Status 404 returned error can't find the container with id bab598c0dc3b79f6cd4b318fbf4f459fade17b289a29b0a2e9a9a8c6040d7a8b
Oct 30 22:51:12.313099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:12.313081    5510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:51:12.750875 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:12.750840    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" event={"ID":"576a17d1-cf8c-46d2-9001-5bde4c055cec","Type":"ContainerStarted","Data":"bab598c0dc3b79f6cd4b318fbf4f459fade17b289a29b0a2e9a9a8c6040d7a8b"}
Oct 30 22:51:13.283058 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:13.283028    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:51:13.757360 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:13.757325    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" event={"ID":"576a17d1-cf8c-46d2-9001-5bde4c055cec","Type":"ContainerStarted","Data":"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"}
Oct 30 22:51:13.757852 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:13.757418    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:15.846374 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:15.846345    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:19.068991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:19.068939    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" podStartSLOduration=8.169985618 podStartE2EDuration="9.068924249s" totalImagesPullingTime="898.938631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:13.86892881 +0000 UTC m=+4082.852835686" watchObservedRunningTime="2026-10-30 22:51:19.068924249 +0000 UTC m=+4088.052831124"
Oct 30 22:51:19.069427 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:19.069305    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:51:19.767601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:19.767572    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:51:20.563601 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:20.563571    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:20.742413 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:20.742384    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:51:20.742564 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:20.742449    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:51:24.047931 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:24.047900    5510 scope.go:122] "RemoveContainer" containerID="a4bb1acd30c548d1ffed54bc7607348dbcef3ac5a304c89a29257b4b9724aac1"
Oct 30 22:51:24.056699 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:24.056679    5510 scope.go:122] "RemoveContainer" containerID="f8b1bf742c807a93ec92f92a04eacb535564b2ccddf73c91932a6861bb5bd925"
Oct 30 22:51:25.057356 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:25.057321    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:29.867281 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:29.867243    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:30.147649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:30.147570    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.361236 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:34.361183    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:38.870241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:38.870204    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:40.064996 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:40.064960    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:40.679363 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:40.679333    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:43.279839 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:51:43.279801    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:51:43.292967 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:43.292935    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:47.783933 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:47.783895    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:52.255084 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:52.255029    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:56.763992 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:51:56.763943    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:01.164153 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:01.164116    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:05.658194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:05.658164    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.169624 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:10.169590    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:13.279310 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:52:13.279276    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:52:14.770162 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:14.770128    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.266285 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:19.266253    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:23.775115 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:23.775043    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.242157 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:28.242125    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:32.768879 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:32.768842    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.241525 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:37.241484    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:40.387544 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:40.387512    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:41.714261 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:41.714205    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.279744 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:52:43.279709    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:52:46.256017 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:46.255951    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.885334 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:50.885301    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:55.924872 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:52:55.923846    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:00.354990 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:00.354958    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:02.116773 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:02.116731    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:53:02.218986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:02.218948    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:53:02.498129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:02.498095    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:02.660825 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:02.660726    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:04.761429 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:04.761393    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:07.117416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.117358    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" podUID="48449120-fac3-47bf-a265-25b85cca8fcd" containerName="payload-processing" containerID="cri-o://e5e7224fca6e8789d70bcba35fb65bd7eb0da1902199cb3f6ab5cd65f2f846b8" gracePeriod=25
Oct 30 22:53:07.219616 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.219573    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" podUID="b934e5de-3037-4684-a46d-39bc3d01e569" containerName="payload-pre-processing" containerID="cri-o://1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b" gracePeriod=25
Oct 30 22:53:07.306108 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.306079    5510 generic.go:356] "Generic (PLEG): container finished" podID="48449120-fac3-47bf-a265-25b85cca8fcd" containerID="e5e7224fca6e8789d70bcba35fb65bd7eb0da1902199cb3f6ab5cd65f2f846b8" exitCode=0
Oct 30 22:53:07.306274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.306176    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" event={"ID":"48449120-fac3-47bf-a265-25b85cca8fcd","Type":"ContainerDied","Data":"e5e7224fca6e8789d70bcba35fb65bd7eb0da1902199cb3f6ab5cd65f2f846b8"}
Oct 30 22:53:07.377479 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.377404    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:53:07.462535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.462505    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:53:07.490190 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.490163    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\") pod \"48449120-fac3-47bf-a265-25b85cca8fcd\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") "
Oct 30 22:53:07.490365 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.490234    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\" (UniqueName: \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\") pod \"48449120-fac3-47bf-a265-25b85cca8fcd\" (UID: \"48449120-fac3-47bf-a265-25b85cca8fcd\") "
Oct 30 22:53:07.490533 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.490507    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume" pod "48449120-fac3-47bf-a265-25b85cca8fcd" (UID: "48449120-fac3-47bf-a265-25b85cca8fcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:07.490684 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.490638    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48449120-fac3-47bf-a265-25b85cca8fcd-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:07.492458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.492436    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj" pod "48449120-fac3-47bf-a265-25b85cca8fcd" (UID: "48449120-fac3-47bf-a265-25b85cca8fcd"). InnerVolumeSpecName "kube-api-access-4f4wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:07.590986 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.590949    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\" (UniqueName: \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\") pod \"b934e5de-3037-4684-a46d-39bc3d01e569\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") "
Oct 30 22:53:07.591194 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.591016    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\") pod \"b934e5de-3037-4684-a46d-39bc3d01e569\" (UID: \"b934e5de-3037-4684-a46d-39bc3d01e569\") "
Oct 30 22:53:07.591272 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.591227    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4wj\" (UniqueName: \"kubernetes.io/projected/48449120-fac3-47bf-a265-25b85cca8fcd-kube-api-access-4f4wj\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:07.591407 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.591379    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume" pod "b934e5de-3037-4684-a46d-39bc3d01e569" (UID: "b934e5de-3037-4684-a46d-39bc3d01e569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:07.593393 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.593363    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn" pod "b934e5de-3037-4684-a46d-39bc3d01e569" (UID: "b934e5de-3037-4684-a46d-39bc3d01e569"). InnerVolumeSpecName "kube-api-access-88nnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:07.692002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.691909    5510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b934e5de-3037-4684-a46d-39bc3d01e569-plugins-config-volume\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:07.692002 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:07.691950    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88nnn\" (UniqueName: \"kubernetes.io/projected/b934e5de-3037-4684-a46d-39bc3d01e569-kube-api-access-88nnn\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:08.312130 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.312092    5510 generic.go:356] "Generic (PLEG): container finished" podID="b934e5de-3037-4684-a46d-39bc3d01e569" containerID="1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b" exitCode=0
Oct 30 22:53:08.312587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.312151    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"
Oct 30 22:53:08.312587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.312179    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" event={"ID":"b934e5de-3037-4684-a46d-39bc3d01e569","Type":"ContainerDied","Data":"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"}
Oct 30 22:53:08.312587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.312252    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss" event={"ID":"b934e5de-3037-4684-a46d-39bc3d01e569","Type":"ContainerDied","Data":"8c1b7f6ec5cdeba4424d0822f70b2d9646b2e3cb55777cad3a688e190e91834d"}
Oct 30 22:53:08.312587 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.312287    5510 scope.go:122] "RemoveContainer" containerID="1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"
Oct 30 22:53:08.314808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.314513    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"
Oct 30 22:53:08.314808 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.314527    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn" event={"ID":"48449120-fac3-47bf-a265-25b85cca8fcd","Type":"ContainerDied","Data":"8124d3eadd75cbc1ece46d2d234a996f63ec64b907a59b2e4614b7aff771313c"}
Oct 30 22:53:08.322649 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.322621    5510 scope.go:122] "RemoveContainer" containerID="1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"
Oct 30 22:53:08.323027 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:53:08.322967    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b\": container with ID starting with 1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b not found: ID does not exist" containerID="1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"
Oct 30 22:53:08.323027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.322995    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b"} err="failed to get container status \"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b\": rpc error: code = NotFound desc = could not find container \"1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b\": container with ID starting with 1a20671403a628cbe9969e70d46da977fa82b68586247b948356bb18970a221b not found: ID does not exist"
Oct 30 22:53:08.323027 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.323015    5510 scope.go:122] "RemoveContainer" containerID="e5e7224fca6e8789d70bcba35fb65bd7eb0da1902199cb3f6ab5cd65f2f846b8"
Oct 30 22:53:08.528521 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.528486    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:53:08.536199 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.536172    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8a1d54-6f585868bc-ccqss"]
Oct 30 22:53:08.641518 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.641480    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:53:08.648173 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:08.648143    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-8a1d54-6cc96cd86b-pdrtn"]
Oct 30 22:53:09.123991 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:09.123949    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48449120-fac3-47bf-a265-25b85cca8fcd" path="/var/lib/kubelet/pods/48449120-fac3-47bf-a265-25b85cca8fcd/volumes"
Oct 30 22:53:09.124416 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:09.124396    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b934e5de-3037-4684-a46d-39bc3d01e569" path="/var/lib/kubelet/pods/b934e5de-3037-4684-a46d-39bc3d01e569/volumes"
Oct 30 22:53:12.046652 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:12.046622    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:53:12.478096 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:12.478072    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:53:13.278936 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:53:13.278900    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:53:13.739440 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:13.739404    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:53:13.739667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:13.739643    5510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" podUID="576a17d1-cf8c-46d2-9001-5bde4c055cec" containerName="maas-api" containerID="cri-o://665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978" gracePeriod=30
Oct 30 22:53:13.808106 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:13.808053    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:14.008311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.008290    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:53:14.161463 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.161428    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") pod \"576a17d1-cf8c-46d2-9001-5bde4c055cec\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") "
Oct 30 22:53:14.161633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.161521    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\") pod \"576a17d1-cf8c-46d2-9001-5bde4c055cec\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") "
Oct 30 22:53:14.161633 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.161569    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\") pod \"576a17d1-cf8c-46d2-9001-5bde4c055cec\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") "
Oct 30 22:53:14.161729 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.161655    5510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\" (UniqueName: \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\") pod \"576a17d1-cf8c-46d2-9001-5bde4c055cec\" (UID: \"576a17d1-cf8c-46d2-9001-5bde4c055cec\") "
Oct 30 22:53:14.161988 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.161955    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca" pod "576a17d1-cf8c-46d2-9001-5bde4c055cec" (UID: "576a17d1-cf8c-46d2-9001-5bde4c055cec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:14.163731 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.163706    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv" pod "576a17d1-cf8c-46d2-9001-5bde4c055cec" (UID: "576a17d1-cf8c-46d2-9001-5bde4c055cec"). InnerVolumeSpecName "kube-api-access-bwvvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:14.163823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.163734    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs" pod "576a17d1-cf8c-46d2-9001-5bde4c055cec" (UID: "576a17d1-cf8c-46d2-9001-5bde4c055cec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:14.163823 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.163791    5510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls" pod "576a17d1-cf8c-46d2-9001-5bde4c055cec" (UID: "576a17d1-cf8c-46d2-9001-5bde4c055cec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:14.263145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.263063    5510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvvv\" (UniqueName: \"kubernetes.io/projected/576a17d1-cf8c-46d2-9001-5bde4c055cec-kube-api-access-bwvvv\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:14.263145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.263095    5510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-maas-api-tls\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:14.263145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.263110    5510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/576a17d1-cf8c-46d2-9001-5bde4c055cec-metrics-certs\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:14.263145 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.263125    5510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/576a17d1-cf8c-46d2-9001-5bde4c055cec-openshift-service-ca\") on node \"ip-10-0-131-231.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:14.346369 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.346330    5510 generic.go:356] "Generic (PLEG): container finished" podID="576a17d1-cf8c-46d2-9001-5bde4c055cec" containerID="665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978" exitCode=0
Oct 30 22:53:14.346866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.346377    5510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"
Oct 30 22:53:14.346866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.346411    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" event={"ID":"576a17d1-cf8c-46d2-9001-5bde4c055cec","Type":"ContainerDied","Data":"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"}
Oct 30 22:53:14.346866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.346458    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb" event={"ID":"576a17d1-cf8c-46d2-9001-5bde4c055cec","Type":"ContainerDied","Data":"bab598c0dc3b79f6cd4b318fbf4f459fade17b289a29b0a2e9a9a8c6040d7a8b"}
Oct 30 22:53:14.346866 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.346479    5510 scope.go:122] "RemoveContainer" containerID="665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"
Oct 30 22:53:14.356131 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.356109    5510 scope.go:122] "RemoveContainer" containerID="665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"
Oct 30 22:53:14.356393 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:53:14.356374    5510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978\": container with ID starting with 665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978 not found: ID does not exist" containerID="665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"
Oct 30 22:53:14.356451 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.356400    5510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978"} err="failed to get container status \"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978\": rpc error: code = NotFound desc = could not find container \"665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978\": container with ID starting with 665976993e6cf46fce8eca46062b35201d1162bb63739ba8c6adc5b200c3f978 not found: ID does not exist"
Oct 30 22:53:14.500076 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.500039    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:53:14.502064 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:14.502039    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8a1d54-6c66c6dd6c-8kksb"]
Oct 30 22:53:15.124274 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:15.124231    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576a17d1-cf8c-46d2-9001-5bde4c055cec" path="/var/lib/kubelet/pods/576a17d1-cf8c-46d2-9001-5bde4c055cec/volumes"
Oct 30 22:53:15.230019 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:15.229983    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:15.492775 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:15.492743    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:16.247492 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:16.247456    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:18.761993 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:18.761942    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:53:22.421313 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:22.421279    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.673983 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:22.673898    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.833985 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:22.833946    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:30.564724 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:30.564683    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:53:40.773867 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:40.773828    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:41.969700 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:41.969668    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:53:43.349179 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:53:43.349142    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:53:56.046760 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:56.046644    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"]
Oct 30 22:53:56.058859 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:56.058828    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-mv2gq"]
Oct 30 22:53:57.123237 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:53:57.123203    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12acde89-b9cc-4b43-8b6e-69e008fd9850" path="/var/lib/kubelet/pods/12acde89-b9cc-4b43-8b6e-69e008fd9850/volumes"
Oct 30 22:54:03.067191 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:03.067155    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:54:13.278883 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:54:13.278848    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:54:20.050067 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:20.050029    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"]
Oct 30 22:54:20.096598 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:20.096563    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tth2d"]
Oct 30 22:54:21.123298 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:21.123266    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa" path="/var/lib/kubelet/pods/598e8c8d-cb8d-4025-8e5b-5f462f9b2aaa/volumes"
Oct 30 22:54:35.634945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:35.634903    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rqcjz"]
Oct 30 22:54:41.051193 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:41.051150    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:43.278995 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:54:43.278958    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:54:46.043224 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:46.043183    5510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"]
Oct 30 22:54:46.089548 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:46.089513    5510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c12f66-q9llp"]
Oct 30 22:54:46.236685 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:46.236657    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6678fd9b7b-bvmtm_ebaabad6-8b43-4e0c-88c5-c34466bbe17f/manager/0.log"
Oct 30 22:54:47.124003 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:47.123968    5510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b50c1a-afa8-4162-8b0d-1582b0c21c01" path="/var/lib/kubelet/pods/89b50c1a-afa8-4162-8b0d-1582b0c21c01/volumes"
Oct 30 22:54:47.645008 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:47.644975    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:54:48.683611 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:48.683567    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6dcc7_4b073d99-dda0-4c19-aa4c-99b181577916/proxy/0.log"
Oct 30 22:54:49.662941 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:49.662911    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd44bb67c-h2qj5_049a576d-50e6-464b-8e8e-f58da7c630eb/maas-api/0.log"
Oct 30 22:54:53.301196 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:53.301162    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tj846_e8987d6e-68b6-48ca-91a9-7d8a706d9b8a/manager/0.log"
Oct 30 22:54:53.605717 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:53.605637    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rqcjz_fe39f822-603c-4dfd-b2bc-2a218e42dffa/limitador/0.log"
Oct 30 22:54:54.891679 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:54.891648    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w8v2q_709595ee-db7b-4df3-be34-fa8a572e5973/discovery/0.log"
Oct 30 22:54:55.113375 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:55.113348    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78776654bd-gpv65_0d51094e-a499-404c-bbba-0e4372d86ffd/kube-auth-proxy/0.log"
Oct 30 22:54:55.556757 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:55.556730    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-525t7_a5c45657-26b2-4455-8d74-5031505f8482/istio-proxy/0.log"
Oct 30 22:54:56.389735 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:56.389706    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8f4kg_3a829d55-09d6-4110-ab64-7b84c761319f/istio-proxy/0.log"
Oct 30 22:54:57.396940 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:57.396897    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ks5ks_0f7f5f7a-9377-4e4c-9399-492dedf7a100/istio-proxy/0.log"
Oct 30 22:54:58.446667 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:54:58.446640    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xpk68_e9798351-651c-4c76-a42e-06819a31734e/payload-pre-processing/0.log"
Oct 30 22:55:00.044326 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:00.044293    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7fxjz_48e0d4a0-2d4f-4769-964f-29120855dee9/main/0.log"
Oct 30 22:55:00.281311 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:00.281264    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-t8b8g_4f613d67-925e-409b-9208-13ecbb1163a9/main/0.log"
Oct 30 22:55:00.789398 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:00.789361    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr_7cafbb56-06c8-47ab-9ec6-d5355209b352/main/0.log"
Oct 30 22:55:01.245753 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:01.245726    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h_a0d8ee34-d09b-4cc9-a870-78846d79306c/main/0.log"
Oct 30 22:55:11.801765 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:11.801720    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6678fd9b7b-bvmtm_ebaabad6-8b43-4e0c-88c5-c34466bbe17f/manager/0.log"
Oct 30 22:55:13.064709 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:13.064673    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:55:13.279225 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:55:13.279187    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:55:14.002123 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:14.002094    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6dcc7_4b073d99-dda0-4c19-aa4c-99b181577916/proxy/0.log"
Oct 30 22:55:14.885140 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:14.885108    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fd44bb67c-h2qj5_049a576d-50e6-464b-8e8e-f58da7c630eb/maas-api/0.log"
Oct 30 22:55:18.138279 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:18.138247    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tj846_e8987d6e-68b6-48ca-91a9-7d8a706d9b8a/manager/0.log"
Oct 30 22:55:18.425242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:18.425155    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rqcjz_fe39f822-603c-4dfd-b2bc-2a218e42dffa/limitador/0.log"
Oct 30 22:55:19.523690 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:19.523659    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w8v2q_709595ee-db7b-4df3-be34-fa8a572e5973/discovery/0.log"
Oct 30 22:55:19.753458 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:19.753429    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78776654bd-gpv65_0d51094e-a499-404c-bbba-0e4372d86ffd/kube-auth-proxy/0.log"
Oct 30 22:55:20.189776 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:20.189748    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-525t7_a5c45657-26b2-4455-8d74-5031505f8482/istio-proxy/0.log"
Oct 30 22:55:20.961589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:20.961520    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-8f4kg_3a829d55-09d6-4110-ab64-7b84c761319f/istio-proxy/0.log"
Oct 30 22:55:21.972253 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:21.972221    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-ks5ks_0f7f5f7a-9377-4e4c-9399-492dedf7a100/istio-proxy/0.log"
Oct 30 22:55:23.027531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:23.027503    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xpk68_e9798351-651c-4c76-a42e-06819a31734e/payload-pre-processing/0.log"
Oct 30 22:55:24.123589 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:24.123556    5510 scope.go:122] "RemoveContainer" containerID="a0a54109e183ea79882c1bbe592312374e5f36fb68178938a2058ff986311c5d"
Oct 30 22:55:24.131956 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:24.131936    5510 scope.go:122] "RemoveContainer" containerID="11c18035513610491bb6e991719515c2b6fcaa804d400ab0cb560ea970991cc1"
Oct 30 22:55:24.289159 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:24.289130    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7fxjz_48e0d4a0-2d4f-4769-964f-29120855dee9/main/0.log"
Oct 30 22:55:24.509982 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:24.509946    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-t8b8g_4f613d67-925e-409b-9208-13ecbb1163a9/main/0.log"
Oct 30 22:55:24.967332 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:24.967304    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b84xwvr_7cafbb56-06c8-47ab-9ec6-d5355209b352/main/0.log"
Oct 30 22:55:25.403862 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:25.403766    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vft4h_a0d8ee34-d09b-4cc9-a870-78846d79306c/main/0.log"
Oct 30 22:55:41.400923 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:41.400879    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:43.279064 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:55:43.279028    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:55:54.185987 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:54.185953    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:55:54.741367 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:54.741336    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:55:55.205648 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:55:55.205560    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.279031 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:56:13.278993    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:56:24.156144 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:24.156110    5510 scope.go:122] "RemoveContainer" containerID="e8adc0de6d929de7e9326db1815a6a72b2478eb361c87e26fccd248d1718b6df"
Oct 30 22:56:32.592241 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:32.586520    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:56:33.128535 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:33.128505    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5844bc79b5-hj279_f10c1f15-c182-4754-a084-cab33f005873/manager/0.log"
Oct 30 22:56:34.525249 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:34.525218    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:34.718124 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:34.718093    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:41.738491 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:41.738456    5510 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:43.279323 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:56:43.279290    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:56:44.558805 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:44.558763    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wr6rh_54348157-31ac-475e-90a1-32b963be27a3/global-pull-secret-syncer/0.log"
Oct 30 22:56:46.382205 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:46.382173    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rtpn_b640ceb8-5397-4ff2-9428-ab03c9cf3474/konnectivity-agent/0.log"
Oct 30 22:56:48.178025 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:48.177993    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-231.us-west-2.compute.internal_0e90b7cbbf3b5bb50f8f3af03ed08b90/haproxy/0.log"
Oct 30 22:56:53.844099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844004    5510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"]
Oct 30 22:56:53.844614 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844591    5510 cpu_manager.go:414] "containerMap: removing container" podUID="b934e5de-3037-4684-a46d-39bc3d01e569" containerName="payload-pre-processing"
Oct 30 22:56:53.844698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844616    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b934e5de-3037-4684-a46d-39bc3d01e569" containerName="payload-pre-processing"
Oct 30 22:56:53.844698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844654    5510 cpu_manager.go:414] "containerMap: removing container" podUID="48449120-fac3-47bf-a265-25b85cca8fcd" containerName="payload-processing"
Oct 30 22:56:53.844698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844663    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48449120-fac3-47bf-a265-25b85cca8fcd" containerName="payload-processing"
Oct 30 22:56:53.844698 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844690    5510 cpu_manager.go:414] "containerMap: removing container" podUID="576a17d1-cf8c-46d2-9001-5bde4c055cec" containerName="maas-api"
Oct 30 22:56:53.844945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844699    5510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576a17d1-cf8c-46d2-9001-5bde4c055cec" containerName="maas-api"
Oct 30 22:56:53.844945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844823    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="48449120-fac3-47bf-a265-25b85cca8fcd" containerName="payload-processing"
Oct 30 22:56:53.844945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844840    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="576a17d1-cf8c-46d2-9001-5bde4c055cec" containerName="maas-api"
Oct 30 22:56:53.844945 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.844856    5510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b934e5de-3037-4684-a46d-39bc3d01e569" containerName="payload-pre-processing"
Oct 30 22:56:53.848018 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.847996    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.896902 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.896875    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjhpl\"/\"kube-root-ca.crt\""
Oct 30 22:56:53.901292 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.901266    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-podres\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.901389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.901301    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-lib-modules\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.901389 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.901327    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjbs\" (UniqueName: \"kubernetes.io/projected/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-kube-api-access-5gjbs\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.901469 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.901448    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-proc\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.901505 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.901482    5510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-sys\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:53.943160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.943131    5510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gjhpl\"/\"default-dockercfg-mpw92\""
Oct 30 22:56:53.943160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.943152    5510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjhpl\"/\"openshift-service-ca.crt\""
Oct 30 22:56:53.944880 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:53.944861    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"]
Oct 30 22:56:54.002953 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.002924    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-proc\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.002977    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-sys\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003051    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-proc\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003073    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-podres\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003099 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003092    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-lib-modules\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003106    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-sys\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003110    5510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjbs\" (UniqueName: \"kubernetes.io/projected/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-kube-api-access-5gjbs\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003145    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-podres\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.003305 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.003214    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-lib-modules\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.057296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.057263    5510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjbs\" (UniqueName: \"kubernetes.io/projected/38f7d2c6-77e8-4b10-b182-b350bc74ddd3-kube-api-access-5gjbs\") pod \"perf-node-gather-daemonset-9mzjp\" (UID: \"38f7d2c6-77e8-4b10-b182-b350bc74ddd3\") " pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.160154 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.160077    5510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:54.458370 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.458344    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tj846_e8987d6e-68b6-48ca-91a9-7d8a706d9b8a/manager/0.log"
Oct 30 22:56:54.527984 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.527957    5510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"]
Oct 30 22:56:54.529688 ip-10-0-131-231 kubenswrapper[5510]: W1030 22:56:54.529656    5510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38f7d2c6_77e8_4b10_b182_b350bc74ddd3.slice/crio-eb047f20c6c4e1db03e7f24b38e4d3466e80dac9d2e040d59fc6c6da95a99ca8 WatchSource:0}: Error finding container eb047f20c6c4e1db03e7f24b38e4d3466e80dac9d2e040d59fc6c6da95a99ca8: Status 404 returned error can't find the container with id eb047f20c6c4e1db03e7f24b38e4d3466e80dac9d2e040d59fc6c6da95a99ca8
Oct 30 22:56:54.844295 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:54.844220    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rqcjz_fe39f822-603c-4dfd-b2bc-2a218e42dffa/limitador/0.log"
Oct 30 22:56:55.427445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:55.427412    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp" event={"ID":"38f7d2c6-77e8-4b10-b182-b350bc74ddd3","Type":"ContainerStarted","Data":"2a165df6cb2724f85ed3ed59037eee01e2c8400b84d8d049a54ca058a9149bec"}
Oct 30 22:56:55.427628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:55.427463    5510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp" event={"ID":"38f7d2c6-77e8-4b10-b182-b350bc74ddd3","Type":"ContainerStarted","Data":"eb047f20c6c4e1db03e7f24b38e4d3466e80dac9d2e040d59fc6c6da95a99ca8"}
Oct 30 22:56:55.427628 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:55.427499    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:56:57.742743 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:57.742703    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/alertmanager/0.log"
Oct 30 22:56:58.142596 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:58.142523    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/config-reloader/0.log"
Oct 30 22:56:58.548000 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:58.547971    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/kube-rbac-proxy-web/0.log"
Oct 30 22:56:58.943296 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:58.943209    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/kube-rbac-proxy/0.log"
Oct 30 22:56:59.343129 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:59.343098    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/kube-rbac-proxy-metric/0.log"
Oct 30 22:56:59.749294 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:56:59.749271    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/prom-label-proxy/0.log"
Oct 30 22:57:00.145435 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:00.145332    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1444bd0b-372f-4db4-9efe-5d330a550ad8/init-config-reloader/0.log"
Oct 30 22:57:01.441414 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:01.441383    5510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp"
Oct 30 22:57:01.550175 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:01.550121    5510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gjhpl/perf-node-gather-daemonset-9mzjp" podStartSLOduration=8.550106994 podStartE2EDuration="8.550106994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:55.533768489 +0000 UTC m=+4424.517675364" watchObservedRunningTime="2026-10-30 22:57:01.550106994 +0000 UTC m=+4430.534013863"
Oct 30 22:57:02.644188 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:02.644154    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wp7wz_ab8d50c8-864a-4758-9ece-62c693d9cffd/monitoring-plugin/0.log"
Oct 30 22:57:03.056373 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:03.056340    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65jnb_76a1da5e-7477-453d-bd89-cbe7f9e7b20a/node-exporter/0.log"
Oct 30 22:57:03.450794 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:03.450704    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65jnb_76a1da5e-7477-453d-bd89-cbe7f9e7b20a/kube-rbac-proxy/0.log"
Oct 30 22:57:03.845009 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:03.844984    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-65jnb_76a1da5e-7477-453d-bd89-cbe7f9e7b20a/init-textfile/0.log"
Oct 30 22:57:10.770445 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:10.770412    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/0.log"
Oct 30 22:57:10.774531 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:10.774487    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/prometheus-operator/1.log"
Oct 30 22:57:11.111242 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:11.111175    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rf5bb_e8b30f97-b15d-4c82-b314-2849193c3a98/kube-rbac-proxy/0.log"
Oct 30 22:57:11.913013 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:11.912984    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67569c9584-zz8p8_1f3aadfa-fbef-4e8e-8ff2-4fb99373d517/telemeter-client/0.log"
Oct 30 22:57:12.308830 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:12.308770    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67569c9584-zz8p8_1f3aadfa-fbef-4e8e-8ff2-4fb99373d517/reload/0.log"
Oct 30 22:57:12.705183 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:12.705106    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67569c9584-zz8p8_1f3aadfa-fbef-4e8e-8ff2-4fb99373d517/kube-rbac-proxy/0.log"
Oct 30 22:57:13.108397 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:13.108368    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/thanos-query/0.log"
Oct 30 22:57:13.280168 ip-10-0-131-231 kubenswrapper[5510]: E1030 22:57:13.280134    5510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-79gfb" containerName="extractor"
Oct 30 22:57:13.504508 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:13.504477    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/kube-rbac-proxy-web/0.log"
Oct 30 22:57:13.900590 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:13.900520    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/kube-rbac-proxy/0.log"
Oct 30 22:57:14.296831 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:14.296806    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/prom-label-proxy/0.log"
Oct 30 22:57:14.692842 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:14.692750    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/kube-rbac-proxy-rules/0.log"
Oct 30 22:57:15.089111 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:15.089043    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d8d48d5b6-7rbd4_a9185dde-8443-434d-9df7-27d7e287b372/kube-rbac-proxy-metrics/0.log"
Oct 30 22:57:16.463160 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:16.463134    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w8v2q_709595ee-db7b-4df3-be34-fa8a572e5973/discovery/0.log"
Oct 30 22:57:16.671935 ip-10-0-131-231 kubenswrapper[5510]: I1030 22:57:16.671896    5510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78776654bd-gpv65_0d51094e-a499-404c-bbba-0e4372d86ffd/kube-auth-proxy/0.log"